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FOREWORD 

This analysis was performed within Advanced Develbprrierit Subprbject Z1176-PN.01 
(Improving the Navy's Sompater-Managed Training System), under the spdrisbrship bf the 
Deputy Chief of Navai Operations (Manpower, Personnel, and Training) (OP-Ol). It 
supports Navy Decision Soordinattng Paper ZH76-PN and its objective bf imprbvirig the 
technical training pipeline. 

The (SNET/NAVPERSRANDCEN Task Group reviewed a draft of this report on 15-lS 
Aprtl 1981. This report reflects any changes emanating frbm that meeting. 

This report documents the results bf an extensive analysis of the requirements fbr the 
Navy*s Computer-Managed Instructibh (CMI) system. The ahalysis was part of the effort 
of a muJtiorganization task group, aimed at improving the CMI system. The report 
Lncludes revisions coming from the Upgrade Task Group meeting held 15-16 April 1981 at 
Pensacbla, Florida. Other members of the task grpUp^ who contributed substahtia 
this analysis, include: Mr. John Hasseh arid Mr. Ed Scheye (CNET N-9)i Mr. 
Bill Ottendorfer _{.CNET Mr. Wilson thbrtias, Mr. Gordon Crawford, and Ms. 

Phylis Salop. (CNTECHTRA N-l), CDR Gloria Holmes, Ms. Diane Kalivdda (CNTECHTRA 
N-63)^ Mr. Ernest Owens (CNTECHTRA N'^3), Mr. Charles Tilly^ Mr. George Businkl, and 
Mr. Larry Shell (MIISA, Memphis Detachment), Mr. Charles Morris (TAEG)y and Dr. 
Charles Lihdahl (NTEC). 



Appreciation is expressed to all members of the CMI System JJpgrade Task Group, 
and to the many individuals at the technical schools and training program management 
levels who provided the necessary information for this report. 



3 AMES F. KELLY, OR. 
Commanding Officer 
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SUMMARY 



Problem and Backgrbdhd 

As i\^e Nayy*s corn[Duter-nianaged mstructipri (CMI) system has ^rbwh in size and 
capabili_ti_es^ CMI schobihbUses have experienced problems with proionged system response 
times (RT) and excessive downtime (DT). These deficiencies slow the^ flow of students 
through the training pipeline. Additidhaiiy, computer system requirements have grown 
beyond those included in the original system design. 

The Chief of Naval Education and Training (CNET) recognized the need to mount a 
direct effort to reconsider the entire scope of system requirements and improve the 
system's bperatidh. Accordingly, CMET and the Na,vy Personnel Research and Develop- 
ment Center (NAVPERSRANDCEN) established a joint CMI System Upgrade Task Group 
to determine the requirements for an improved CMI system. 

Obj ectlves 

The objectives of this effort were: 

1. determine current and near-term CM! system instructional requirements arid 
provide preliminary instructional system specifications. 

2. To develop alternative system upgrade approaches that would satisfy the 
instructional requirements. 

Approach 

A CNET/NAVPERSRANDCEN task group was established in October 1980 to plan for 
an instructional system requirennehts analysis, to bbtairi the requirements informatjon, 
NAVPERSRANDCEN developed and adrriihistered questionnaires to and conducted inter- 
views with personnel _at_CMI and hbn-CMI technical training "A" schools, Chief of Naval 
Technical Training (CNTECHTRA) training Program Coordinators, CMI systern manage- 
ment personnel. Navy iristrUctibhal prbgrarh developers and evalua,tors, and computer 
personneLfrom the Management Ihfbrmatlbh and Instructional Systems Activity (MIISA), 
Memphis Branch Office. The survey results were integrated into information and problem 
summaries for each brganizatibnal elerhent in the CM! system. - By usjng the information 
summaries and making cbhtact with MIISA personnel, CMI system upgrade alternatives 
were developed. 

Results 

1. The major CMI system deficiencies perceived throughout the CMI community are 
prolonged RT and excessive system DT. 

2. CMI schools are satisfied with the kinds of CMI capabilities available but are 
generally concerned with system reliability. 

3. MIISA has taken immediate short-term actibris tb relieve RT and DT problems, 
but this has altered the schools' CMI capabilities sorhewhat. 

4. A need for ADP support of grb^p-paced ihstructibh was identfied and the 
instructional requirements were specified. \ 

\ 



5- tnstractlonai reqairements for ah expanded capabilities CMl system were specifi- 
ed, 

6; Six CMl upgrade alternatives were specified with rough cost estimates provided 
for each; 

7. Given the constraints of limited time and resources^ the recommended upgrade 
alternative consists of developing the system to cbhtihue the current CM! fuhctidhs. 
However, reliability should be improved (no more than 5 percent DT at the learning 
center), and automated data proicessihg (AD_P) support should be added for group-paced 
instructibh. No DT [:>eriod should exceed 10 minutes duration Until switch to a backup 
mode is accomplished nor should there be more than fdUr int.>rruptiohs per day per shift. 

RecbiTimendatibhs 



1. CNET should continue the CWI upgrade effort to resolve RT and DT def icjencies^ 
The upgrade effort should include adding ADP support for group-paced instruction. This 
shbrt-terrh upgrade effort should result in a system with no rnore than 5 percent DT at the 
learning center equipment cluster, even when the host computer is down. 



2. As part of the upgrade effort, MHSA should direct a systems anaj^^^ identify 
precise /" uses of system malfunctions, and to provide computer specifications for the 
CMl and rtbP supported group-paced instruction requirements. 

3. CNET should coqrd^rate all CNET Computer programs that affect data file 
structures and system operation on the CMl computer. 

4. CNTECHTRA should ensure that Training Program Gbbrdinators are knowledge- 
able about CMl reports for thetr courses. 
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INTRODUCTION 



Backgrourid 



The cufreht Navy computer-managed instruction (CMI) system was l^sed on a 
pfbtbtype system design developed by the Navy Personnel Research and Development 
Center (NAVPERSRANDCEN). Instructional requirements used to specify the current 
system were determined during the early 1970s. Since that time, the system has been 
expanded to perform functions not considered in the original design, and the computer, 
which vvas origihally dedicated to support CMI, is how being used to support a number of 
additional information systems. Meanwhile, more courses and students were added until 
Navy CMI is presently serving approximately 9000 students daily. _ft_{na.jor new 
requirement to provide automated data processing (ADP) support for group-paced courses 
is also being considered. 



Pro blem 

During the last year, the CMi schoolhouses have experienced problems with deterior- 
ated system response time (RT) and excessive downtime (ST), making it difficult for 
students to have tests graded and receive study assignments in a timely manner. This 
inefficiency slows the flow of students through the training pipeline. Although a number 
of short-term actions have been directed toward solving the:ie problems, a continued 
expansion of the system may again lead to degraded operations. 

Both the Chief of Naval Education and Training (CNET) and the Chief of Naval 
Technical Training (CNTECHTRA) recognized the heed to mount a direct effort to 
improve the system's operation and to reconsider the entire scope of its instructional 
requirements. As a first phase of this effort, it was necessary to determine the 
requirements for an upgraded system and to provide the specifications for alternative 
instructional systems. It was assumed that, shpijld a major systeinh redesign effort be 
warranted, the selected design alternative would be implemented with relatively limited 
funding and personnel resources. 

Objective 



The objective of this elfbrt was to identify current and near-term CMI system 
ihstructiohal requirements, and to provide preliminary ihstructibhal system specifications 
that would satisfy the requirements and ensure that planned exjpansioh, in both size and 
capabilities, could proceed without causing degraded system operation. The information 
cbhcerhihg alternative system designs is intended to aid CNET in deciding the optimal 
approach lor upgrading CMI, taking into account both the benefits of added capabilities 
and improved system operation and the cost of additional hardware and personnel. 



APPRdACH 

3-Qiat Task Group 

A joint task group was estabHshed in October 1980 to develop a plan for determihirig 
instructional system requirements and for providing supporting information for CNET for 
use in selecting an approach for upgrading the CM! system. The group was cbmprised of 
staff from the following organizations: CNET (N-2, N-7, N-9, 013); CNTECHTRA (N-1, 
N-63); the Management Information and Instructional Systems Activity, MemphisjMIISA), 
Nvhich operates the computer; the Training Analysis and Evaluation Group (TAEG); the 



Navy Traihihg Equipment CenteF (NTSC); and N AVPE_RSRANDCEN._ Cb-chairmen for the 
group were from CNET N-9 and NAVPERSRANDCEN. A_ list of specific task group 
members attending the initjal meeting is provided as part of Appendix A, which lists all 
Navy information sources contribating to this effort. 

Pru^ce dure 

To elicit information regarding system problems and user requirements, the task 
group surveyed all components of the instructional system ihcluding: the schools 
(including those that are not sup^Dorted by CWl)^ nianagemeht^ MIISA, the instructional 
program development centers (IPDCs), and research orgahizatiohs. The survey was 
conducted by means of a series of questionnaires developed by NAVPERSRANDeEN. 
CNTECTRA Gode N-1 (the CMI system manager) served as the point of contact for the 
distribution ana return of questionnaires from the many schoolhouses, trami^ng program 
coordinators (TPGs), and other staff brgahizatibhs within the Naval Technical Training 
Command. Questionnaires for bther organizations were distributed individualiy and were 
frequently completed during personal interviews. Most questions were open-ended so as 
to ensure the broadest possible cbverage of system problems and user need. 

The task group aisb_requested extensive data on each course from available Navy 
information systems. These data were used as background for interpreting certain 
responses to the questionnaires. 



Analyses 

The survey results were integ^rated into summaries representing the consensus at each 
maior echelon and organization comprising the CM! system. Accordingly, information bh 
problems and needs are presented from the perspectives cf: _ the technical training 
s_chgblhbUses, training program management at GNTEGHTRA, CMI system mahagemeht at 
CNTECHTRA, the computer organization (MHSA), and the instructional program develop- 
ers/evaluatbrs (IPDC/CiMET/CNTEeHtRA). Technological opportunities fbr CMI were 
identified based on survey information obtained from research brgahizatibhs and personal 
contacts with experts from participating task group organizations and other civilian and 
military agencies. 

Information from the survey was used as a basis for developing instructionaj functions 
specifications for (1) an upgraded CMI system and (2) a system to provide ADP support for 
group-paced instruction. These functional specifications, in turn, were used as a basis for 
developing alternative system colhfigUratibhs. RbUgh estimates of costs for each 
configuration were provided by MIISA. 

SUMMARY OF SURVEY INFORMATION 

Information from Technical Traihihg ScHbalhouses 

Questionnaires were sent tb all schools where CNTECHTRA-controlled AI and A_P 
courses were being taUght. Schools already supported by CMI were asked to identify 
problems with the existing system, tb rate the utility of available CMI reports, and tb 
suggest. instructional system modifications that might increase CMI effectiveness. A copy 
of the CMI course questionnaire is provided in Appendix B. 

The questionnaire for schools that djd Hot have CMI courses focUsed bh pKissible ADP 
support fbr twb general areas: (1) administrative requirements (develbpihg ahd using 
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records a. d preparing reports)^ and (2) testing requirements (scbri rig tests arid processing 
arid storing th-^ results). Iristructibris for the iqiiestibririaire asfced the resjDbriderit to 
develop a school cbhsehsus fbr each qUestibn. While the riature bf the qUestibPs prevented 
elabbrate statistical analyses^ the bbtairied irif brmatibri was readily suniriiarized arid is 
prov'ided in the following paragraphs. 

CMI ^uppo rted Schoo[s 

Personnel^ from nine C Mi-supported courses responded to the questionna,ire: Aviation 
Machinists Mate (AB), Aviation Fundamentals (AFUN), Avionic Technician (A V), Basic 
Electronics and Electricity (BE/F' locations). Interior Communications Electrician (iC)j 
and Radioman (RM), Responses d ; tered greatly as to the frequency and severity of the 
ratings on the two major operational problems (Rt and DT), and the ratings on other- 
deficiencies. One of these other c.eficiencies, GMI course coding, was mentioned with 
some degree of frequency, but it vas never rated as more than a minor problem. The 
remaining problems listed were few in number, course specific, and also rated as minor. 

Personnel frbrii the CMI schcbls were asked. to list any new functions or capabilities 
that shouldlbe cbrisidered for the CMI system. In general, there was little commonality 
among these suggestions. Several respondents suggested that it would be helpf uJ to_ have 
faster RTs fbr irifbrmatibri that would be used duririg an Academic Review Board (ARB), 
arid that sbrrie of the existing CMI repnorts used for that purpose^ particularly the Student 
Respbrise Histbry^ cbuld be rriade mbr^^ Several respbnderits indicate, a desire to 

cbhtihue usirig the N AVPERSRANDCEN-deveJbped Inceriti ve Charts^ which were receritly 
evaluated during an bperatibrial test. Most bf the rerriairiirig cprrimerits_ pertairied tb 
rela.tively minor revisibns in certain reports (e.g., list Navy and Maririe Corps studerits 
separately, or flag 3ob Oriented Basic Skill (3 OBS) students). 

Each respondent was asked to rate the utility of each of the 18 reports that are 
rqutmely available from the CMI sj^stem. Separate ratings were made for learning center 
personnel and for administrative and managerial personnel. Detailed responses regarding 
report utility are provided in Table 1. As shown, although the utility ratings for many of 
the indj vidua, 1 reports were quite j^riable, each report was ra,ted as Very Useful by at 
least three oi the schools^ As noted previousjy, several respondents suggested minor 
report revisions, but indicated nc real need for additional kinds of reports. 

The three Academic Remediation Training (ART) programs, located at Navy Training 
Centers in San Diego, Great Lakes, and Orlando, are also supported by the CMl system. 
However, their needs differ considerably from those of conventional CMI courses, because 
of their combined i.idividualized and group-paced form of instraction and the testing 
procedure used to select students. The ARTs were similar to other CMI courses in their 
ratings of slow RT and excessive DT as the major system deficiencies. The ARTs also 
cited several problems arising frorri_ their unique form of instruction (e.g., a need for 
multiple stjjdy assigrirn;erits)._J'he_A RTs used few of the routine CN^'l reports. However^ 
both the ARTs and CNTECHTRA Code 017, manager of the ARTs, indicated a .substantial 
rteed fbr repnDrts arid surinrriaries that a '•e riot provided by the current system. The desired 
surrirriaries require access tb derribgraphic data that are riot record^ by the currerit 
system, br tb perfbrriiarice data iri subsequerit courses br iri the fleet. This is ari example 
of the need tb provide an autbrriated data rilaragerrierit capability that extends acrbss the 
training/fleet pipeline and is centrally managed. 

In summary, questionnaires from schools now on the CMI system describe an 
overwhelming concern with slow RT and excessive system DT. While other problems do 
exist, they are. far less important to the schools. In general the CM! schools are satisfied 
with the current set of CMI Instructional functions; 



Table 1 

Ratings of Routine CWI Reports by Gourse Persbririel 



Learning Center 
Personnel 

Very Moderately Not 

Report Usefal Useful Useful 

Student Learning Guides 8 1 . 0 

Daily eWI Student Progress 

Reports 9 0 0 

Deficient Progress Report * 1 * 

Accelerated Progress Report 15 2 

Trend Analysis Report 1 2 6 

Extra Study Report 3 3 3 

Learning Center Status Report 1 6 2 

CDP Distf IbUtidn 9 0 9 

School Rosters k 2 3 

Graduation/ Drop Rosters 0 ^ 5 

Projected CMI Graduation 

Report 0 3 5 

Student Perforrnance Sarrirriary 0 3 6 

Monthly Student Flow Report 0 1 8 

Module Distribution Report 0 2 7 

Response History Print 3 1 5 

Student Status Matrix 1 ^ ^ 

Item Analysis Report i 1 7 

Module Peffbrmance Analysis 

Report^ 0 0 8 

his report was not available in the AV course. 



Adrriiriistrative arid 
Managerial Personnel 

Very Moderately Not 
Useful Useful Us efai 

1 4 ^ 



9 
6 
5 
2 
3 
7 

n 

9 
7 

6 
6 
3 
6 

6 



0 
2 
2 
5 
3 
2 
2 
0 
2 

2 
1 
1 
5 
3 
2 
3 



0 
1 

2 
2 
3 
0 
3 
0 
0 

3 
2 
2 
1 
0 
3 
0 



MI Supported Schools 

Non-CMI school respondents were asked to indicate which of their records * reports, 
and analyses might be handled by a computer, and to estimate the num^ber of man-hours 
per month now devoted to each activity. Total estimates ranged from 5 hours per month 
to 885 hours per month. Estimates tended to vary with the average number of students on 
board {Am), but there were still substantial variations--from .1 to t.D hours per. student 
on board. Part of this variation was probably due to real differences in the reporting 
Fe^uirements for different brphlzations. Another part was due to differences in attitude; 
some respondents listed activities that might be quite awkward for a computer, while 
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others failed to list activities that are required in ail courses and that are handled 
routinely by computers. 

In general^ the responses serve to documeht a fairly substantial need for comp^ 
support in hbh-CMI courses. Some of this need is already being met through the useS^of 
local data processing facilities such as Scantron scoring devices coupled with microcorRk- 
puters. TAEG is presently developing the functional specifications for such a facility to 
be used by the Apprentice Training program at NTC, Great Lakes. The^TAEG specificaJ 
tions would prohably be applicable to all Apprentice training pr;o^grarns. Computer 
supfHjrt of the kind now [n existence ^.^rids to be highly variable and nonprogrammatic. 
Most of the perceived needs for ADP are not being met at all. 

Each non-CMI respondent was asked to provide inf^qrmation on the kinds and 
frequency of tests they administer. The fr'equencies reported were extrenne[y variabiej 
ranging from one short test every few weeks to approximately sjx tests each day. The 
higher frequencies were typical in the individually-paced courses, but they were also 
found in some of the group-p)aced courses. Much of this variability is due to differences in 
the intended function of the tests. In some courses, it appears that the sole purpose of 
the tests was to provide occasional ranking of students. In others, the tests were used to 
facilitate learning, to ensure mastery of training objectives, and to evaluate the quality of 
instruction. Generally, tli^ majority of non-GMI courses administered informal tests or 
quizzes two or three times a week and formal tests about once a week. 

_ The kinds of questions used oh these tests also differed widely from course to course. 
The majority of items were multiple-choice and true-false questions that could easily be 
scored by automated rneahs. A considerable number of matching and fill-ih items of both 
a numerical and alphabetical nature were also used. Many of these items could probably 
be converted to a forrriat suitable for automated scoring. In fact^ the possibility of such a 
fbrrhat coriversibh was rhehtibhed by several respbhdehts. Finally, there were several 
cburses in which many tests could be handled thrbugh the use of special eqUiprrieht (e.g., 
bpticdl character readers) arid special ADP prbgrarris (e.g., prbgrarhs similar to the one 
used fbr scbririg teletypirig testing iri the RM "A" cburse). 

Attitudes toward automated test scoring appeared about evenly divided for and 
against, with the more favbrable attitudes occurring among courses with Tar^ge AO^ 
fact, one of the most frequently mentioned responses for rejecting automated scoring was 
the belief that it would not be cost effective (or simply not worth the trouble) in the very 
small courses. There wa,s also a feeling that automated test scoring would di^rnmish the 
rapport between student and instructor. Why this relationship between ra^pport a 
automated scoring should be more of a problem in a sma^ll cqur^se than^ in ajarge course^i^^ 
not clear from the questionnaire responses. _ f^inally, jt appear e^d that spm^ 
reacted negatively towa,rd the possibility of autorn^ated test scoring sim because they 
did not want GMI, even though the questionnaire stressed the point that automated 
scoring is not synonymous with 6MI. The fact that automated test scoring of one kind or 
another is already being used in several of the courses is indicative of the general level of 
acceptance, if not outright demand, for ADP support. Attitudes toward automated 
scoring and the use of a computer to 5tore test results tended to be correlated, but the 
relationship was far from perfect. Some respondents wanted automated scoring for 
quizzes, but did not want to retain the results. Other respondents wanted to store data 
from tests, even when the tests would not be scored by automated means, it was clear 
that many of the respondents had not perceived the possibility of storing test data as a 
means of satisfying the schools' continuous reporting requirerhent. Finally^ several 
respondents indicated an interest in generating tests autbrriatically from a stored pool of 
test questions. 



By chance, questionnaires were sent to several schools with A2j C\j arid C7 courses. 
These courses had small AOBs, and their responses tended to be similar^ to those of the 
smaller Al courses; that is, a number of therh felt that their requireriierits were riot 
sufficient to justify a computer-based system for either ad mirii strati ve support or test 
scoring. Courses of this kind tend to be physically clustered into groups with related 
subject matter. In many cases, these clusters are at major training sites (e.g. ^ Meriiphis br 
San Diego); in other cases, they are relatively isolated (e.g.. Mare Island). Iri either case, 
it may be suitable for the courses within a cluster to share elemerits of a cbrilputer 
system, even though ADP support could not be justified for each iridividual course. 

Although questionnaires were not sent to the Appreritice Training courses, informa- 
tion from GNET^ indicates that the total AOB is large (about 2300), that testing 
requirements in a batchmode are substantial, arid that' automated test scoring is either 
available now (e.g.. Great Lakes) or is plariried for the riear future (e.g., San Diego). The 
TAEG effort to develop fuhctibrial specificatibris shbuld apply to all apprentice training 
sites, as mentioned previbusly. 

In summary, the questibririaires frbrii rion^CMI courses document a fairly extensive 
need for ADP support^ everi if it were limited to functions that are obvjous, sucji as 
scoring muitipl^chbice test s^ arid were provided only for courses that actively request 
ADP suppbrt. The tbtal ADP requirement for aU non-CMI courses is probably more 
extensive^ but comprehensive documentation of these require^ments calls for a ^^m^^ 
extensive analysis with greater resources than were available during the preparation of 
this report. 

Information from Training Pro gram Coordinators at eNTEGHTRA 

Questionnaire^ were sent to all TPGs on the CNTECHTRA staff to bbtain Iriput frorii 
mid-level management. The TPCs were asked to consider the possibility bf ADP supp^brt 
for group-paced courses and to indicate the kind and amount of suppbrt that rriight be 
needed^ TPCs with CfvlJ courses were asked, in addition, to list deficiericies bf the 
existing CMI system, suggest improvements, and indicate their Use bf standard CMI 
reports. To date, only 12 responses have, been received. Since those that were received 
tend to be sirnilar to those obtained from the individual schbbls, the absence of a more 
cornplete return may not be serious. 

Seve/al TPGs provided exterisive lists bf requirerherits for non-CWI courses that 
might be rriet by a computer; others lirriited themselves to brief geher^al discussion^s. A 
majority bf the responses were positive with respect to both autornated adjpinistrat^^^^ 
support and testirig. There were sbme reservations about ADP support for small courses 
or for courses that rely alriibst exclusively on perforniance testing. However, one TPei 
who manages a sizable training program, felt that "there is nothing to be gained with 
cbmpHiter suppbrt in this area." One TPC discussed the possible use of_computer-driven 
trairiirig devices, the few TPCs who commented on how ADP support might be provided 
favbred local cbmputers operated under local control. 

Mbst of the CWI TPCs cited DT as the single serious problem with CMI. They offered 
rib suggestions for additional capabiUties^ The two TPCs who responded to the questibri on 
report utilization listed only the^ Performance Summaries and the Monthly Student Flow 
Report. This low level of utill^tion suggests that few reports are useful at this level bf 
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management, or that the tPSs are hot aware of the reports that are available. Since the 
TPe for the BE/E schools uses the GMI reports extensively in making management 
decisions, tt is apparent that the GMl data can be of great value at this level. 

Information from the CM! System Manager at CNTECHTRA 

Extensive information was obtained from the CMI system manager's ^office at 
CNTECHTRA (N-8) regarding: description of current fuhctibhs, current system deficienc- 
ies, new functions desired, areas, requiring empirical, verification to assure system 
optimization, and requests for CM! data. The extent of this ihf brmatibh dbtaihed from 
the system manager reflects the fact that he becomes involved in many prdblerh calls 
from users, handles all requests for CMI data^ serves as the interface between the training 
community ajid the computer prgahizatioh^ and is respbhsible to mahagerrieht for system 
bperati bh. The description _ of current fuhctibhs was used in evaluating respbhses frbm 
other cbmpbhehts bf the CMI cbmrhuhity> but was tbb extensive for direct inclusion in this 
report. The interested reader can find detailed descriptions of system functions (e.g., 
student registration^ training path selcctibh, test scoring and feedback, remediation and 
prescription, studeht progress trackingj^ and course cbmpletion) in Section 1 of CMI User 
Manual 0 (Guidelines f br the use of the Navy Computer-Managed Instruction System). 

System Deficiefictes 

The system manager listed the foUowing items as deficiencies: 

1. Excessive DT and the inability to assess DT accurately at the individual school 
level. 

2. Slow RT and the inability to measure RT at individual schools. 

^ 3. back of fiexibility in test item format (e.g., limited to multiple choice). 

back of a resource allocation and scheduling capability for assigning students to 
limited equipment or space in certain courses. 

The inclusion of the RT arid DT problems bri the system rriariager's list rib dbubt reflects 
the continued concern expressed to the N-8 office by the iridividual schbbls bver this past 
year. The needs for greater flexibility iri test item Jbrrriat arid a resburce allbcatibri 
capability have been noted by N-8 for several years. The severity bf these deficiencies 
has become more apparerit as they have beeri ericburitered by rribre instructional program 
develbprrient efforts. 

The follbwirig iteriis, although hot listed as deficiencies, were listed as areas requiring 
empirical verificatibri tb assure system optimization. 

1. Criteria f or selecting an instructional delivery system. 

2. Guidelines for designing the optimum learning center. 

3. Guidelines for instructional design in computer-managed courses. 

Because these three a,reas are relevant to a broad scope of training problems, vhe detailed 
N-8 comments are included in Appendix C. 

N-8 also discussed the problem of CMI course coding. Inefficient br iricomplete 
testing of the Initial course codes results in incorrect codes being implerrierited bri ari 
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operational basis, this leads to a loss of time for both students and instructors, since the 
instructor must determine the coding prbblerri and then get it corrected. N-8 recom- 
rnends, and the task group supports development of an on-ljhe coding capability that could 
have built-in debugging steps to assure adequate: testing of course codes prior to use with 
students. Such a capability would be essential in the absence of a centralized coding 
facility. 

There are also problems in meeting special requests for CMI data. During the period 
1 October 1980 through 1 March 1981, N-8. received about 20 special requests for CMI 
data from CM! users in eNTEGHTRA and_from other organizatidhs such as NAVPERS- 
RANbCEN, TAEG, GNET, and CNO. Each of these requests involved a .special 
programming effort by MIISA personnel and, because of conflicting demands, required 
abduO0-4b days to satisfy. The structure of the CM! data base is such that summarized 
information cannot be obtained, easily from the systerri. This limitation is due to the 
seqdfentiai method in which CM! data are stored and is inherent to the current system 
architecture. 

Areas for Potential System Imprbvemeht 

The following functions were suggested as additions to the system: 

1. A resource allocation and scheduling capability for assigning and tracking limited 
resources such as equipment and lab stations. 

2. A training p^th selection capability that can adjust to limited resources by 
making alternative assignments. 

3. A capability for predicting course completion that can adjust for variations in 
student assignments caused by limited resources. 

Development and evaluation of procedures for using alternate test item formats 
such as constructed response questions. 

5. Development of ApP support for group-paced instruction including test scoring, 
information reporting, and student progress management. 

- 6. Ability to summarize and extract performance iriformatibri readily for both 
individuals and groups. 

7. Ability to relate individual test iterris to objectives, so as to improve the 
evaluation of training and the use of feedback from the fleet. 

8. On-line testing. 

9. Cornputer-generated tests. 

10; An on-line capability for course coding, with automated debugging [5rocedures. 

Although the problems and suggestions expressed by the system manager are similar 
to those expressed by school personnel, two major differences in perspective are worthy 
of note. First, the system manager places a greater eniphasis on the use of an ajlocation 
and scheduling procedure that might eliminate bottlenecks due to limited resources. 
Second, he perceives a greater need to improve the guidelines for course dejign^ It should 
be obvious to the reader that, no matter how efficiently a GMI system operates, student 
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progress, arid achievement will be deficient if either the iristructibrial materials or the 
course design is defective. 

In sumrnary^lhe system manager feel5 that the major probJems with the current 
system are slow RT arid excessive DT^ and that the major system limitatibhs are the lack 
of test item f lexibility^ the lack of a resource alldcatibri and scheduling capability^ 
inadequate CMI course design guidelines^ and ah inadequate data rhahagerherit capability. 
Without better ihf brrriatibri about design alternatives^ arbitrary guidelines may be 
systerhatically degrading the effectiveness of each course added to the system. 

Information from MIISA^-The Computer ^rg^aTiizatioh 

Information regarding CMI system problems was obtained f rom MIISA during personal 
interviews with staff and from a questionnaire* The questions focused on problems with 
Q^J_^Q*^ "^'l^i^.possibje solutionis the extent of support for systems other than CMI, and 
information regarding the cost and scheduling of alternate CMI system design options. 

MIISA recognises the DT probleiTis out that this problenn is mucn greater in 

the classroom thari at ^he ^^ntral com^puter sjte. The central computer is oj5erating well 
within its contractua.i maximum of 'ive percent DT^ In fact^ recerit analyses indjcate the 
Mainframe computer is down only one to three percent of the time.^ Although thiere are 
obviously other problems in the chain between the central computer and the equipment in 
the learning centers that contribute to the DT experiend^^d by students and instructors, 
isolating the precise location and cause of these problems extremely difficult. Similar 
diagnostic problems exist for the slow RT. Recently, so m^^riy changes have been, made to 
improve RT that it is impossible to specify the exact effect of each change. MIISA 
recognizes the possibility that the computer system is virtually saturated with its present 
student and course load. 

Three factors that may be causing problems will be discussed: 

1. The loading of ajzprripUter that was originally dedicated to CMI for jobs _such as 
military payroll arid MILPERSIS may well be ihterferihg with system operation. The full 
list of data prbcessihg functions supported by the Memphis computer^ other than those 
directly associated with CMI^ is provided below: 





MILPERSIS 


Computer Directed Training System 




STDM 


Corps of Engiheefii 


• 


NITRAS 


Persdnriel 


0 


100 Application Programs, 


NMPC Accounting 




NAS Memphis 




Military/Civilian Payroll 


Resources Management System 


• 


3-M 



Fr^om this list, it js obvious that there is a si^^ load beyond that imposed by CMI. 

Any difficulties created by competing demands on the computer are exacerbated by the 
absence of formal priorities for the different pf'Pjressin^ systems. A 
frequently asserted that the C^MI evaluation (EVAt) program has^high priority, there are 
certain situations in which EVAt defers to the other systems such as MItPERSlS. 
(CNET should prbvide formal guidance for establishing system priorities.) 

2. Because\ of the design of the EVAL program itself, all inforrhatibri must be 
processed sequentially. As a xesult, when the prbgram becomes saturated, data are 
backed up in the buffers and RT increases. Contacts with the MIISA staff indicate that 
the 



resolution of this limi^^ would require the redesign of the EVAL program, if not the 
entire system software. 

3. cm functions now assigned to the computer go far beyond those ori^^^^^ 
envisioned. When the system was first developed, it was modeled after a design for a 
relatively small systetri. While efficiencies were built in the full Navy system, the full 
gamut of kinds of cobrses to be supported (e.g., highly individualized, courses with 
frequent testing and refnediatioh assighmehts, or large numbers of tests to be scored all 
at once, and automated teletypihg test scoring (Radioman "A" school)) were not 
considered in the design structure. Although MIISA has improved the software program to 
accommodate these additiohal functions, a major system redesign may be necessary both 
to resolve system problems and to permit an orderly addition of other courses or 
functional capabilities. 

Failures throughout the system leading to DT at the terminals were caused Jpr a 
variety of reasons. The reasons for these failures, and the approximate percent of DT at 
the terminal, are listed below: 



Reasons 

1. Terminet failures, all sites. 

2. OPSCAN failures, all sites. 

3. DT due to WILPERSIS failures. 
"U. DT due to communication problems. 

5[ DT due to computer hardware failures; 

6. DT due to operator error. 

7. DT due to equipment/software changes. 

_ _ It should be noted that MIISA has recently made a numb:er of.chahges to alleviate the 
RT and DT problems. Current analysis shows an average RT of 2 to 10 secbhdSj although 
• someJlTs are still as long as 2 minutes while tapes are being changed. This occasional 
long RT indicates the continued existence of an RT problem. Additionally, the RT 
improvement that has been achieved has come about at the removal of some of the 
schobTs capabilities. For example, schools are no^ sonnewhat limited to when they can 
register students. If th[s limit on registration prevents students from beginning study on 
the day they amve, it may cost the training command the remainder of the first day when 
the student gets to a CMl school. While the changes can improve immediate system 
functioning, MIISA recognizes the heed to reach a permanent solution that will permit 
full-system function at the schoolhouse level and allow planned system growth without 
further deterioration of system bperatibh. 

A major support problem facing MIISA involves the lack of course cooing resources. 
Although the original CMI cbUrse implementation schedule has slipped considerably^ 
present course coding requirements exceed the MIISA capability. A major issue involves 
deciding whether tb assign coding responsibility to MIISA on a centralized basis or task 
the IPDCs tb assume coding tasks. At present, coding quality is maintained when 
performed at MIISA-Memphis by experienced coders. Problems of course coding occur 
frequently when performed outside MIISA due, in part, to the high turnover of non-MlISA 
coders, who are relatively inexperienced. 
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Infbrrhdtibh from the Area of Instructional Program Develbprrierit and Evaluation 



Ihfbrmatibh frbhri this area bf ihstrUctibrial prpgranni develog^meixt jahd jeyaluatibh_^^^ 
bbtaihisd from interviews ahd^ q^^ administered tb CNTECHTRA (Code 016), 

CNET (Code N-9), and the IPDCs located at San Dlegb arid Great Lakes. (Refer to 
Appendix A for names of specific individuals contacted at these brgahizatibris.) Individu- 
als were asked to indicate the iimitations of the CMl system and the steps th^it shbUld be 
taken to improve the system, with particular concern fbr problems in the areas bf CMl 
course design and development. They were asked specifically to indicate their perceptibh 
of instructional requirements for ADP support of group-paced instruction. 

_ ^J^APA^b^_^J^st f reque system limitations was that of the constraint on use 

of different types of questions for tests. Several comments were obtained regarding the 
nee^l for constructed response cjuestions for proper achievement testing. Although most 
respondents felt this wa^ a senous limitation on the design of courses, it was not a 
universal perception. Another needed capability repeatedly mentioned was automated 
scoring of performajice testing. The comments ranged fro^ criticism about the 

current pass/don't pass provision for performance test scoring to one suggestion for 
scoring drawings of electronic circuits. As might be expected, the instructional 
developers did not discuss the cost factors associated with this form of test ^cqrjng^ since 
their concern is with development rather than with cost accounting. Testing needs were 
also indicated by comments regarding the necessity for computer generation of tests from 
a pool of questions. Test security was cited as the major justificatton, although ease of 
test item revision and distribution of test materials would also benefit, if the test 
generation system were conducted on-line. Again, no mention was made of the costs 
associated with this capability. 

Several individuals suggested the use bf CAI for remediation purposes. This* was 
mientioned in the context of having the system prbvide guidance to send a student to a 
nriicrbprbcessbr that would prbvide ihstructibh bh:a_particular set of objectives. This 
incbf pbratibh of CAI within the rriariagerrieht rble of CMI appears particularly appropriate 
given the prblif eratibh of small cbmputers within the training community. Prbper 
admihistrative cbhtrol bver the CAI lessbhware would be heeded tb assure the integrity of 
the test materials. 

One suggestion provided by respondents was that CWI might be used to prbvide 
remediation for individuals in a group-paced course. This suggestion is worth pursuing if 
the costs for the CMI remediation would be comoensated for through reduced student 
dittrition. An implicit problem in this mixing of g.oup-paced instruction and CMI is the 
mi:dng of mastery-based testing used in CMI with norm-referenced test procedures 
gerierally used in group-[>aced instruction. 

Again, the need for a resource allocation system was mentioned frequently. In 
technical training schools where there is a limited amount of training devices, use of the 
training devices by Individuals can create student-flow bottlenecks if the use of the 
devices is not optirnized. The allocation and schedaling system would serve this purpose. 

Although course coding was not mentioned as a serious problem, there were 
comments regarding the need_to improve the communication interface between the course 
designers and the coders. There is a con :ern that centralizing coding at MIISA might 
exacerbate coding prbblems through further deterioration in the designer/coder interface. 

The IPD persbhheL indicated that the CMI Item Analysis RepKDrt and the Module 
Performance Analysis Rep>ort are used with a fair degree bf frequency in their course 
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validation efforts. Other GMI reports, sUch as the Student Learning Guides, Stadent 
Projgress Reports, and the Response History Reports, are sbmetirnes used^f or small-group 
tryouts of instructional materials. In regard to reports^ a number of comments were made 
about the difficulty in obtaining special data requests. This delay in obtaining special 
data printouts is due to the requirement for WIISA to program each special data request. 

In response to the question about required ADP support for group-paced instruction, 
several respondents indicated the general need for batch scoring of a group of test sheets 
in a short (15-20 minutes) time with the results provided back to the instructor as a listing 
of students and test scores. The most detailed description of guideliaes. for automated 
support for g^roup-paced instruction, which was provided by CNTECHTRA Code 01^6 
(Ms. Diane Kalivbda)^ has been distributed to other organizations and has been independ- 
ently cited a_few times as an indication of what ]s needed to support group-paced 
instruction. This document is provided for detailed study in Appendix D. An additional 
document that gUided the task grouj? team in their analysis of group-paced instruction 
support was Draft CNET Instruction 15^0.2, subj: Testing and Management of Student 
Achievement. 

to cbhtrast the type of support needed for group-paced instruction as oppposed to 
individualized instruction, the following summary is presented: 

While test scoring for CM! is done on an individual student transac- 
tion basis, scoring for group-paced instruction should be done on a 
batch basis, with all students in a class having the same test scored 
at the same time. Instead of individual student learning guides being 
produced by the system, as is now the case for CMI, the computer 
should provide the instructor with a list of student names or SSNs^ 
the test scores, an indicatioh of whether each student ^ssed or 
failed, and a listing of objectives failed for each student. The 
instructor could then either orally make a remedial study assignment 
to the class for particular objectives 6r post studj^ assignments for 
the list of objectives, and the students could determine their own 
individual remediatibh leqiuremerit. In many ol the larger courses, 
providing the feedback tb the student may be even less of a concern. 
The prirriary concern is in making sure that the tests are scored 
rapidly and that the data are stored for subseq»ient analysis for 
management purposes. The development of the computer, program 
and school administrative procedures that would perm^^^^^ 
mastery testing/remediation on an^ individual student within a grbUp- 
paced cburse^-and still maintain class progress- -remains Jo be dbhe, 
althbugh the desire for this capability is mentioned repeatedly. Frbm 
the combined responses from IP D personnel, the task grbUp derived 
some general requirements for group-paced ADP support. These are 
provided in the report section on Instructional Fuhctibn Specifica- 
tions. 

^garding Technological Opportunities for CMI 



Information regarding technological bppbrtUhities for CMI was obtained from inter- 
views and personal contacts with experts frbrh Navy training and research organizations^ 
other military and civilian agencies, and private corporations involved with computer- 
based ihstraetional technology. Fbr the purposes of this summary, the technologipaj 
opportunities to be presented. include those technologies that were judged to have a_ high 
probability of successful implementation ihtb a large-scale, operational, computer-based 
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training system. However, becaase of the extensive harriber of available hardware 
components, each having a prescribed set of capabilities arid electrbriic/mechariical 
requirements, no attempt was made to catalogue these devices. 

This samrnary will include a description of the technology involved, its advantages 
and disadvantages, and examples of where the technology has previously been appligd^ 
even if only on an experimental basis. 

Ori-line Testing 

Ori-line testing is the administratibn and scoririg of tests through the use of 
iriteractive Computer termijials. The student perceives the test question, usually frbrri a 
Cathode Ray Tube-(CRT) display, and enters the answer by rrieans of a keyboard, although 
light/sound pens arid touch panels are used bccasibrially. Ori-line_testirig permits easy 
revision of questions, automated data eritry^ arid test scbririg. For a large centrally 
rhanaged, nation-wide CMl systerii such as the Navy's^ bri-lirie testing offers (1) the 
ppteritial beriefit of uriifbrriiity of questions across the sarrie course offered at multiple 
sites, (2) immediate system-wide ihnplemehtatibh of revised questibhs^ and (3) a wider 
rarige bf questibri format types beyond simple multiple-choice and true- false. The 
disadvaritages of bh-lihe testing include (1) a greater equiprhent requirement, since a 
studerit would be tying up a terminal thrblighbut each test adrriihistratibh, and (2) a 
greater requirerrieht for corrij-uter membry, because the computer wbuld how have to 
store the eritire t^xt f or each questibh and all answer bptlbns. A detailed analysis of test 
requirerrierits arid equipment capabilities is necessary to weigh the increased equipment 
arid computer costs against potential savings due to reduction of hard-copy test 
reprbdiictibh, rnalllhg, and revision costs. The technology is readily available and ha^s b^^ 
used in hUrherbus cbmputer-assisted instruction (CAI) systems. Any cost effectiveness of 
a computer-based instruction (CBI) system [s genera,lly not attributed to the use of on-line 
testing. An additional benefit for certain applications of on-line testing is the use of 
adaptive testing procedures to reduce eacji jO^ividual test length. ^Q-line adaptive 
testing is presently being implemented in the recruiting centers across the armed 
services. This application, however, js highly suitabl^e for axJ^apti testing, in that very 
few tests are on the system (development of each adaptive test is ext^rejriely ri and 
time consuming), and each test is administered to extremely large nurnbers of _testej:s. 
Development of i^dajptiye tests for a.n operational CMI system that uses rnany tests in a 
variety of courses to assess student proficiency is not considered feasible. NTEG is 
presently developing a Programmable Aiding/testing Learning Module (PAtM) that could 
provide on-line testing if software were developed. The PALM provides a video display 
and an audio capability. While intended .primarily for GAI support, it could be integrated 
with a CMl system. 

A related technology that perrriits better stijderit data eritry without the expeh^se bf 
interactive student terminals is another fbrrri of electronic test-ariswer_ input._ CNET is 
presently pursuing this technolbgy_iri a fbrrri that evblved frbrri the Test Input Device that 
was. xJ escribed by H^mbvitch (1980) arid was ecbribrriically justified by Swbpe and Morris 
(19^0). Currerit plaris for the prbtotype input device develbprrieht are cbhtaihed in a 
report published by the TJaval Weapbhs Center (1981). this fbrrh bf student test data 
eritry will replace the paper answer sheets and the existing bptical scanners now used in 
Navy CM! with ari expected training cbst avbidahce bf apprbximately $^.1 million over a 
6-year period (Swbpe & MbrriSj 1980). 

the develbprhent in recent years of a wide variety of data input devices and 
corriputer terrriinals such as the PALM device that can use on-line testing presents an 
equally wide variety bf bptibhs fbr the CMI system. However, extreme caution must be 
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taken prior to equipment acquisition to ensure that system costs decrease with no 
reduction in training effectiveness, or that systerri costs increases are compensated for 
with equivalent irnprovemehts in training quality. 

Constrocted Response Testing 

Constructed response testing involves the use of test questions that rec^^^ 
student to recall and produce_an answer, as comj5ared to onjy recognizing an answer, as in 
multiple-choice questions. Typical cdhstructed response questions are short answer fill- 
in. Constructed response questions are certainly not a new technology for the world pf 
instructional development. The technological problem has been h to_implement this 
type of question effectively on a computer-based instructional system. Even now^ most 
CAI programs use only multiple-choice or true-fajse questions. The implementatibh 
difficulty involves developing suitable. scoring procedures to account for misspelling and 
correct answers Using alternate words. Software development is extremely complex a 
specific to subject matter. Instructional development procedures caU for recall type 
questions for sorhe types of objective^, but the present GMI system prevents tiieir use due 
to the systerh's limited question fornna,1: capabiiity. By using an electrbriic test-answer 
input device, either on- or ofJ-l[ne, and the appropriate scoring procedures and computer 
prbgramming, the CMI system could employ this type of question. Lockhart, Sturges, 
Van Matre, and Zachai (1981), in a study of the effects of test item forrriat on learning 
arid knowledge retention, found that valid-constructed response qUestioris resulted in 
greater 2-week retentic»n of information than did multiple-choice tests. However, there 
was a cost of longer trctining time. There may be applicatibris iri certain courses where 
the greater amount of retention would- be worth a limited increase in training time. 
Analysis of particular subject matter is necessary to reveal those content areas where this 
form of testing would be cost beneficial. It should be noted that_ use_ of constructed 
response testing, or other technology or equipment, may be appropriate on a limited basis 
for a CBI system and would not require total system implementation. The CBJ system 
would have to be designed from the bUtset tb accommodate these additional capabilities 
even on a limited basis in order not tb iriterfere with overall system operation. ^ T^^^ is 
not to say that on-line testing, or even CAI^ cbUld not be done effectively away from the 
em system, with input to .CMI being manually performed through administrative 
transactions. 

Computer Aided AUthbririg 

Computer-aided authoring is simply the use of the computer to facilitate instructibri- 
al material text aUthbririg and editing. This "authoring" does not refer to course codings 
which could also be conducted on an on-line basis. The text authoring techriology 
combines interactive computer terminals with special computer software permitting on- 
line gerieratibn of textual material. This type of authoring system, if combined with ari 
iristrUctibrial computer system^ allows for the jmmediate transmission of rriaterials frbm 
an aUthbririg site to an instruction site. Course text material could be revised with equal 
ease. With IPDCs located in different geographic locations, an on-line aUthbririg system 
would facilitate CM] course materials development and validation at other training sites. 
The authoring systems can also stand alone since, they dori't have tb have a shared data 
base. A more feasible use with operational CMI would Utilize the authoring system to 
generate on-line testing capability, with authoring limited tb test questions and answer 
options. 

A notable e)®mple of an authoring systern developed within the Navy community is 
the system described by Braby arid Kiric^id (1981). A form of on-line authoring system for 
coding a CMI course should be devclbped to permit aUtorii_ati_c debugging of course code 
prior to use by stUderits. Havirig the coding performed centrally at MIISA would improve 
cbdirig accuracy. ' 



The primary disadvantage of a computer-supported authoring system is the computer 
system expense. This cost should be compared with potential savings expected from 
improved authoring efficiency and rnaterials validation and production. For course coding 
at the MIISA site alone, the_ cqmjDUte would be considerably less and probably 

well worth the addition of that cap>ability. 

The computerized instructional system^ as with the other 

technological developme^nts discussed, represents r opportunities fo^'^pplyi^g 

state-of-the-a^rt instructional and hardware tec^^ these do come with a 

relatively high [nitjal cost. They^re not yet teasibje for system-wide implementation. 
However^ should the decision be made to upgrade the GMl system by means of a major 
system redesign, then in-depth consideration should be given to implementation of these 
technologies. It may be that the full benefit of these technologies may not accrue to the 
training community until a major GBI development effort is conducted, as is how being 
contemplated for the Defense nepartmeht for the next decade. 

INSTRUCTIONAL FUNCTION SPECinCATIONS 

This section summarizes the requirements, for the current CMI system^ for ApP 
supported group^paced instruction^ and for a CMI system with expanded capabilities. The 
required iristructibtial functions were derived by analysis of the survey irif orrhatibri 
regarding deficiencies in the existing system and desired capabilities for CMI arid ndn-CMI 
courses. 



While current CMI users express a major need for operational improvement, they do 
hot express any significant need for change in instructional functions. Accordingly, no 
change in required CMI system functions is proposed. The interested reader can see a 
detailed function description for the existing CMI system in Secti^on 1 of CMI User Manual 
0, Guidelines for the Use of the Navy Computer_M^naged Instruction System . 

The task group did develop CMI system operational standards that should be appli^ 
to any upgrading of the current system. These standards are contained in the following 
list. 

1. To ensure proper school functioning, the CMI system shotrfd provide, at the 
school level, the functions of test scoring and return of test results with 95 percent 
reliability, even when the MIISA host computer is down. The fohctioh of providing 
prescriptive remedial study assignments should also be included, if possible, within 
resource constraints^ 

2. The RT, from insert of answers to receipt at the printer of a typical 20-lihe 
student learning guide, should be ho longer than 30 seconds, _Learhihg guides may be a 
maximum length of ^0 lines and have a maximum associated RT of W seconds. 

The task group believes that if the current CML3ystem_ were modified to meet the 
preceding staiidards, the major system prbblerhs of RT arid DT would be eliminated. 

Because of theexterisive adrriiriistrative and testing requirement for non-CMI schools, 
it is riecessary to develop the instructional furictibri specifications for an A DP system that 
would support grouj>paced iristructibh. In addition, the technological advances are such 
that any review of CMI functions should provide function specifications to derive sys^tem 
upgrade alternatives that would provide support for group-paced i_nstructi_qn a.nd expanded 
CMI capabilities. To provide a structure for describing function specifications, an outline 



was developed that lists all of the general fanctions carried ^Ut any training system, 
irrespective of how, or to what extent, they are accomplished.. The function dutline is 
presented below, and descriptions ol activities conducted in the function are contained in 
Appendix E. Two examples of ocitlines used as aids in devejopihg the present one were 
provided by Lintz, Tate, Pflasterer,^Nix, Klem, and Click (1979) and Wicheli, Morris, and 
Swope (1 980). 

1. Instructional Delivery 

2. Instnjctionai Management 



Student registration 
Training path selection 
Student/instructor data input 
Test scorihg/feedback 
Remediatioh/prescriptibh 
Student progress tracking 
Management ihf brrhatioh reporting 

3. Instructional Support 

Instructibhal prbgram development 
Training prbgrarh validation 
Resource allbcatibn 
Instructibhal system security 
Histbrical - '^'^H keeping 

This butline organizes the following descriptions of function specificatibns fbr ADP 
slippbrted group instruction and for an expanded GMI system. 

ADRSupported GfOUfi=^paced (ADP-GP) Instructional Function Specificatibhs 

. Group-paced instruction in Navy technical Jraihing is conducted in multiple schools 
located at numerous bases across the country. Each schobl may have classes in more than 
one building. Classes may vary in size from less than 20 tb rhbre than 500. Also, the 
method of instruction within a class may vary ranging from grbup-paced^ through group- 
assisted self-paced, to fully individualized for pbrtibris bf the course. The courses will 
vary from mostly knowledge-oriented tb mostly perfbrmance-briehted cours 
result, the ADP-GP system must be flexible enbugh tb accbmmodate extremely varia^^^ 
instructional requirements. The function specificatibhs prbvided here handle this flexibil- 
ity by estiablishing upper limits tb cbver the majbrity of the cases but stiU witt not ^b 
prohibitively expensive. Fbr each general function, the ADP-GP function specif 
involving the learning ehvirbhrheht is presented. The specification i^sjhen followed by a 
section regarding cbrhputer implications. The comments in the computer impH 
paragraphs shbuld be viewed as reasbned suggestions but not precise design requirements. 
Computer design suggestions are based on the precept that processing should be kept as 
close tb the user as possible to minimize RT and DT^p^^^^ 

systems analysis is necessary to specify exactly where in the ^coniputer s 
particular process should occur. Thus, these comments should be considered only as 
suggestions to facilitate costing of preliminary system upgrade alternatives. 

Stude nt Registrati xua 

Specification . Students should be registered onto the systerri at the school site into a 
desigrated course. There should be provisions for entering nbrmal student data, including 
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identifying information, demographic data, arid aptitude ter.t scores, as well as hbhrbUtihe 
information, such as student's participation in^ special prugram (e.g., job C3riehted ISasic 
Skills OOBS)). It shouid be possible to register students either individually or in groups 
^h/_^yg^„^'^? the training day or af ter normal school hours, the student should be able to 
begin tra,inihg during th^e sarne day of registration. Friday evening registration should bis 
possible to permit a Monday training start. 

Computet^ Implication. Since much of the student data may already be in this or 

^t^^^_Ay_^^^J^?? ^hjU^^^^"^^ system^ should be capable of completing or verifying student 
registrati^on by interfacing with other available information systems. It should also be 
possible to change student dat^ incorrectly entered. Data on active students should be 
maintained on a computer at the geographic site, with records established at a centrally 
connected mainframe. 

Training Path Selection 

Specification . This function will assign students to one of several different course 
patterns that define different curricula within a single coars^. (These patterns may be 
designed to accommodate differences in branch of service, rating, or prospective 
assignment.) The system should be able to assign a student to more than one instructional 
module at a time. The instructor shouid be able to override a pattern, module, or lesson 
assignment, assuming appropriaxj administrative approval. The instt'ictor should be able 
to assign a student to night study. 

Computer Implication . Since most group-paced courses are hot divided into instruc- 
tional modules, as in individualized instrujtioh, this function, at the module level, would 
not be expected to be very active. However, the capability should exist for those courses 
desiring it, arid definitely should include the capability of assigriihg students to night 
study. 



Student Data Input 

Sp^c4f icatibn . Student data (responses to test questions) should typically be entered 
through paper machine-scorable answer sheets. Students would use pencils to mark the 
desired responses to the multiple-choice questions; Electronic test-^answer input devices 
could efficiently be used for student data entry, if the school organized their testing 
schedule so all classes would not be taking tests at the same time. 11 class tests were 
sta.ggered, a class of 25 students could use the devices and take their tests, have those 
data entered into the instructor terminal, and then pass the devices on to the next class of 
students for their use. This staggering would permit the greatest student/test-device 
ratio possible, thereby minimizing training equipment costs. 

Computer Implication . Peripheral equipment at the instructor level should permit 
entry of data on paper answer sheets. In some locatiqr^s, this would require ^ high-speed 
reader such as a Scantrori, arid an optical lirik to match up with the student electronic-- 
test-answer input devices. Not all schools or classrooms would have to have both entry 
methods. 

Instructor Data Input 

Fuhctibh . Ihstructbr data wbUld typically be entered through the micrbtermirial at 
the classrobm or school level. Appropriate codes should be available, so that the 
instructor cbuld ehtx^r adrhinistrative^ instructions via answer sheets or through the 
eiectioriLc. test-answer input devices* This permits the instructor to enter information or 
query the system without requiring the instructor's presence at the terrhihal. 
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Cbmputer Implication , No major impiicatiori other than the cautionary note that 
codes should be established for certain specified adrriihistrative transactions, as is now 
possible with the existing CMI system. 

Iest3coring/Feedback 

Specification , The systerri should permit the scoring of lar^e nambers of tests at 
essentially the same time. At the classroom level, the instructor should be able to ask for 
the scoring of a particular test^ enter as many as 100 answer- sheets for that test and 
within 15 minutes, receive a hard copy listing of classroom I.D., date, an alphabetical^ or 
SSN list of students (choice of school), each student's total test score (number of items 
correct), an indication of failed objectives for each student, and average class score. 
Tests would consist of_Up to 100 multiple-choice items with each item coded to specific 
learning objectives. Test scoring for classes with more than 180 students would be 
permitted to take longer, although no slower than 300 tests scored and data returned per 
hour* Each classroom would not be expected to have its own test scoring system but, at 
.least, each building containing active classes would have the scoring capability. At the 
school's dptibn, study/test time could be entered on a class basis for each test. 

If the computer located at the geographic site was not operating but the instructor 
terminal was operating, there should be a provision for the instructor to physlcaUy get a, 
scoring key from a secure location, enter the scoring key, arid then enter student answer 
sheets. This process would only provide the instructor with the number of items correct 
and average class score, but it would permit contiriuatibri bf instruction as a back-up f or 
those times when the geographically-located computer was down. At the classroom, this 
function should be available 98 percent of the tiriie. After the cbmputer was back up, the 
answer sheets would again be entered to make sure data gbt intb the system. 

Computer Implication . At each school building site, there needs to be at least o 
microterminal that would be operated by a Navy instructor. Peripheral equipment should 
permit high-speed acceptance of machirie^scbrable answer sheets (such as a Scantroh) and 
provide hard copy printout. Instructbr sigri-ori wbuld call up the appn^priate test-answer 
key and objectives list for. scoring at the Ibcal Mainframe or minicomputer. The scoring 
key would include a pass/fail criteribh fbr each objective. Student responses and test 
results would be processed arid stbred in the Mini^ with reports sent back to the instructor^ 
Test results would then be sent tb the central Mainframe. Response histories wo ul^ be 
maintained at the Mini while students are active. Class files should be retrievable either 
in alphabetical or SSN lists. As students complete the course, data would be stored on 
tape at the Mini'for use during course validation.. The Mini would have to be structured to 
handle multiple requests fbr test-answer files, since several schools or classrbbrris might 
submit scoring key requests at the jsime time. Files would be needed to store class 
study/test tirries, if those data were entered. 



Remediatibf 



Specif icat4on . The system should provide remediation arid a study prescription if 
bbssible; at the minimum, -it should provide the insfrucm,Mth ah indication of the 
objectives failed by each student. The instructor could then riiake his own remedial study 
assignment, either orally or by posting an assignment list for each bbjective. TyjDicaily, 
the instructor would check his summary test result listing arid then critique the test for 
the class as a who[e. Instructors could assign remedial tests as desired^ although these 
tests mijht be submitted at night study. The class listing of failed objectives woidd 
enable the instructor to emphasize in a lecture those objectives failed by a large number 
of students. 
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Computer Irriplicatibh . The_systerri should make a determihatibh of remedial 
assighrrieht or study prescriptiph. The system would match student answers against the 
scoring Pcey arid print but failed bbjectives as a minimum feedback. System structure 
shbuld permit tests to be scored arid data to be entered for ah individual student but bh an 
excisptlbh basis. Immediate response for a single student test entry would not be 
expected. 

Student Progress Trackin g 

Specification . Progress tracking in group-paced courses is largely a matter of 
knowing how rnany days the student has been in a course, test scores, objectives 
passed /failed, amount of setteck or advance time, a^d^t^ for coarse compietioh. it 
should be possible for tjie instructor to query the system to determine the student's course 
progress in terms o^ number of days in course, days to coarse completion, number of 
setback/advances and their duration, objectives passed/failed, and test scores. The 
instructor should also be able to obtain information regarding study or test time in hours, 
on a class basis, if it has been entered. This information should be available oh a class 
basis in alphabetic or SSN listing or for individual students. Response time should be 
within 10 minutes throughout the day. 

Computer Implication . Instructor queries would be made at the classroom level. 
Data should be retrievable for a class of students or individuaUy, Response time should be 
within 10 minutes throughout the day. Study/test time (hours) for mbdules/tests would be 
available only if entered for the entire class by the instructor. 

Management Ihfbrmatibh Reporting 

Specificatibh . Ihfbrmatibh repbrtihg ih the traihihg ehvirohmeht involves providing 
data tb the studeht, ihstructbr, schbbl, mahagemeht, ahd cbUrse developers. In the group- 
paced systerri, the studeht would get his ihfbrmatibh frbhl the Ahstructor. The instructor 
shbuld b^^ able tb bbtaih data «is a result of ah administrative query at the classroom 
terminal. Tl^e ihstructbr reports, available bh a class or individual student basis, should 
include the follbwihg ihfbrmatibh: cbUrse pattern status matrix (sequence of modules), 
scores bh specified tests, bbjectives passed/failed, missed tests for authorized/unauthor- 
ized reasbhs, number of setbacks or advances, number of assignments to night study, cjass 
standing on a test or f or the course, and days to graduation. The data elements should be 
available at the instructor's option. ^ 

At the school ^2ve^, the same inforr*^ ^y^jl^^'^ ^.T^^^i"^ 

administrative query for separate classes ^ _f o _ t he e retire sch^oh The data should be 
requesti^d by category (e.g., setback or advafce repor^, class standings (average scores for 
students in a class, average scores for a cla^ on a specified test, or cumulative average 
score for each student), students assigned no night study, students in specified course 
patterns). Additional school reports should be available regarding equipment or instructor 
scheduling. All of these reports should be available to the Curriculum and Instructional 
Standards Office. 

At a higher management level, data should be available summarized for classes at 
each school or summarized for several schools. Mariagemeht reports should be provided 
during off hours with no immediate response required. 

Course deyelbpers shbuld have available hot bhly summarized data for individual 
students and for classes of students^ but also data regarding student respbhses tb 
particular tests or test items^ test/study tirrie, fbr cbUrse validatibh purpbses. Course 
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developer data would not be obtained from a terrriirial in a school^ but frdrri a terrriirial at 
the course deveioprnent office. Because response time is hot critical for the course 
developer, their data requests would be met on a batch job basis^ since tape files might 
Ptave to be searched. 

Computer Implicatibh , Given the data summarizing requirements of the system, a 
greater amount of file storage must exist at the Ideal site level than is presently 
available. The implicatioh of this is that rribre software will reside at this level; 
Admihistrati ye procedures should be established that would allow software modification's 
only at the Mainframe level to be transferred to the Mini. For the management requests^ 
data summaries might be developed at each site and then passed as a unit to the 
Mainframe. Since the historical records for all students completing a course are kept at 
the Mainframe, only a search of active files at the local sites would be; nece^ary to 
provide a comprehensive system data summary. A fundamentai implication of the data 
requirerhehts is that there would have to be a data base management capability of greater 
flexibility than exists in the present CMI system. 

Iristructibhal Program Development 

SpeGlfication . Instructionaljprogram development includes both course authoring and 
validation. For this system, there should be an on-line course authoring (coding) 
capability to develop and delxig course files^ This form of authoring does not include 
authoring of instructional text or even test questions, but only the computer coding of the 
course. Centralized coding may be the optimal approach for this function. The school 
should have the capability to request changes to course coding, assuming appropriate 
administrative a, pprov^l. Active course file code changes should typic^lly_i>e .performed 
during nontraining hours. Some consideration could be given to providing IPDCs with ah 
on-line authoring system, independent of the ADP-GP system, if resources were available. 

instructional program validation involves that eyaluatibri necessary to assess the 
quality of the instruction. Course develbper5 shbuld be able to extract ihfbrmatibri 
regarding student answers to specific test questibris, sumrriary scores f br bbjectives/tests, 
or courses. A program should be available to assess student attitudes tbward the cdurse 
and school. Item analysis and the student performance summary reports should be 
available to developers biit would not require immediate availability. 

Computer Implicatiori . Oh-lihe cburse coding at certain sites would have to be 
accommodated at the local level and shbuld include certain debugging procedures to 
ensure that adequate testing of cburse cbdes takes place. Another option would be to 
have a centralized coding capability. On-line text authoring on independent computer 
systems would have rib effect whatsoever oh the ADP-GP system. An attitude assessment 
prbgrarri would brily involve cbdirig bf ahbther test that had assessment items as questions. 
Review of the attitude assessrrierit would be done through an item analysis program. 

Training; Program V alidation 

Specification . Training program validation focuses on assessing the overall course 
effectiveness as opposed to determining module objectives or test validity. This means an 
overall concern with how well the course is m^eeting training objectives, in terms bi fleet 
performance and cost effectiveness. For overall system evaluation, schools will be 
entering into the system as j^rt of _t^he Resource scheduling fur ctton, the data about the 
numbers of students, instructors^ and eqcri^meht purchases. The school should then be^ 
able to request a report providing information about bverali operation costs, cburse 
development co^ts, training costs per student day, etc. These system evaluaJ:|on reports 
would not be available immediately, but should be available within oriejyork week. 
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Computer ImpUc^^Oon . While mucii of the required irifbrmatioh. is already bh 
computer systems, it will be necessary to build groQp file structures and prbgr^mp that 
c:ah acquire, store, and process this information. Again this will require a sbphistica ted 
data management capabUtty. School data would be kept at the Mini, while the overall 
system data would be stored only at the Mainframe but would be accessible at the school 
ievei, assuming administrative approval for the request. 

Resource Allocation 

Specification . Resource allocation in a group-paced, course should involve the 
scheduling of students, instructors, and equipment. For the ADP-GP system, the primary 
accounting requirement would involve equipment inventory listings. For a few courses 
using a limited number of training devices, a scheduling system should be available for 
determining which students have or have hot used the device. A report should be 
available to the instructor regarding who has or has hot completed training with the 
device. 

Computer Implication . The resource allocation system files would need to be 
maintained at the Mini level, but searchable from the classroom. Certain reports 
regarding usage of very limited equipment cbUld be automatically generated on a 
specified basis (e.g., daily). 

Instructional System Security 

Specification . Integrity of the school tests is crucial and is. Maintained pri^ 
using the computer for scoring. Since a backup method for test scoring involves having 
hardcopy scoring keys, these keys must be kept in a secure tocation at the school. 
Additional test security should be provided by having the order of response alternatives 
for multiple-choice questions altered for different administrations of the test. For any 
one class of students taking a test, however, the same scoring key would be used, so the 
instructor might need to observe students duri^ng_ testing to prevent students Jrom 
cheating. Instructor codes would have to be used to ensure that only instructors enter 
information into the system. This pa.rt of the system could be secured by making the 
microtermihal off limits to the students^ Computer IB/Password codes should be used to 
make sUre that unauthorized use of the computer for non-school purposes does hot occ^r. 

Computer Implication , Computer security schemes have to be established to limit 
access only to authorized individuals. A system for altering the sequence of response 
options would have to be developed. This could be done at either the Micro or Mini levels 
of the system. A computer monitoring system should be developed to examine the kind of 
usage occurring at each Micro to determine whether it is being used properly. 

Historical Record Keeping 

Specification . Historical records should be kept readily available on the total system 
for at least 2 years and stored archivally thereafter. Data available on a quick-response 
basis would be limited only to active registered studehts. Data on students graduated or 
attrited from the school would only be available through an administrative request 
honored on a batch, time-available basis. 

Computer Implication . The historical records for students should be kept on the 
Mainframe for 2 years. They should be stored in summary fashion that vADuld still perm^^ 
comparison of course objective data with fleet performance data for an individual student 
for system evaluation. Other data regarding the effectiveness of the system (training 



developmerit, equipment, and instructor costs) should also be sumrharized and stored at 
the Mainframe site for system evaluation purposes. With the greater number of students 
on the system (over 10,000 AOBs estimated), sizable facilities for data storage are 
necessary. An elaborate data management capability is probably necessary to make these 
data available on a timely basis. 

Expanded CM! Instructional Function Specification 

If analysis shows that a redesign effort is necessary to fully resolve the Rt and DT 
problems, then the redesign might profit from the addition of capabilities not presentiy on 
the CMI system. The expanded CMI system^ then, is comprised of functions now satisfied 
by the existing system, plus those functions expressed as desirable by operational ^ 
that would benefit the training communities^ as indicated by research or application data. 
The specifications provided in this report will fdcus on those additionaj functions that will 
••expand" the range of system capabilities for CMI. Since some of these functions have 
yet to be demonstrated as being truly cost effective, their actual implementation on^a 
system-wide basis should await those cost data. However, if the functions are ever^to .be 
implemented, they must be considered in any initial redesign analysis so proper file 
structures, memory^ and software can be provided. It is feasible that a particular 
function could be developed in the central computer (by dedicating file structures) but not 
fully programmed. If resources were available in the future, the programs could then be 
built on a test basis. If small-scale tests showed the function profitable, the function 
could be implemented system-wide. This development approach would avoid purchasing 
equipment that would not be used operationally. Since some of these fun ctibhs are little 
more than concepts, it has not been possible to provide a .detailed specification. 
Accbrdirigly, the resulting estimates of costs for the expanded CMI system should only be 
viewed as rough approximations. Management can at least have ah order of magnitude 
esimate about the cost of the designated expansion. 

The expanded CMI system can be envisioned, at the schbolhouse, as the current CMI 
system with expanded methods for data entry (including optical scanners, electronjc test 
input devices, and microterminals), greater flexibility f or data manipulation, and greater 
capabilities for course and system evaluation. AJ<ey design precept to make the expanded 
system operate with the minimum DT and fast RT is to keep computer processing as close 
to the user as possible. While the present system_desigh does not specif y all processing at 
the terminal level, there is an obvious heed f or a greater amount of test evaluation at the 
local site level, along with an increased heed for processing and storage at the central 
site. One could expect the expanded system to be more costly to design, develop, and 
implement than a redesigned system providing only the current CMI functions. ^ ^ A 
questioh, however, is the extent to which the costs of developing an expanded system are 
greater than those of developing a redesign of the current system, if the expanded system 
is hot significantly more costly than that of a redesigned current CMI system, then its 
development might be warranted. A thorough systems and economic analyses would be 
required to answer the question definitively. Figure i is provided as a pictorial 
representation of the expanded CMI system to be described in the following pages. 
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TECHNICAL TRAINING SCHOOLHOUSE 



Ftgare Pictorial represehtatibh of expanded CMI system. 



Instracttoriai Deltvery 

Specification . CMI is defined as a system, in which ihstrUctidhal delivery occurs off- 
line the computer systerri. In ah expanded CMI system^ however, there should be the 
capability to deliver adjunct ihstructibri_ through connected micrdprocessors. These 
processors would be connected to the CMI system to share summary data but all 
instfactibhai delivery would occur via software on the microprocessor. This instruction 
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amounts to CAI and Would be very Hmhed in scope. Generally, it would be provided brily 
in a ftiw courses and only for those course objectives that require the dynamic processing 
capability of a computer. An example of such a learning objective suitable for this type 
of instruction is one that involves understanding the process of variational analysis of 
current flow in a complex electronic circuit. Without detailed knowledge of course 
objectives requiring this type of instruction^ it is not possible to provide an accurate 
system "specification.*' What is necessary for the exparxJed CMI system is the prbvisibri 
for the interface between the expanded system and the adjunct microprocessor. 

Computer Implication . Ah interface would be necessary between the system 
computer and the microprocessor. Since there would be a variety of subject matter in 
different courses, there should be a standard interface specification to permit this 
capability td_ be implemented as the requirement is identified and materials are 
developed. File space in the expanded system would probably hot be large enough for 
storing summary performance data for individual students. 

Student Registration 

Specification . For the expanded system, student registration wiU be very si, milar to 
the current system. It should be designed so that a student can register at any time and 
irhrriediately begin interacting with the instructional system. Additionally, there s^^ 
be a simple procedure for transferring registration information on a studerit from one 
course to another. There should be a provision for storing nonroutine information (e.g., 
student's participation in a special program such as 30BS) an^d an^ automatic program for 
prioritizing student entries into a course, i_f_there [s a tecklog. These functional additions 
are of a modest na.ture and wojjld^ develop and low in cost, if done 

during a major software redesign effort. They might have significant payoff by improving 
student flow through the training pipeline. 

Com pute^ Jmplicatl on . The registration functions would be easy to incorporate 
during a redesign effort and would not significantly add to the processing requirement. 

Training Path Selection 

Specification . The major addition for this function involves the capability for 
reassigning students to scarce training resources. The system would Have tb rnoriitbr 
student assignment to modules and interact with the resource allbcatibh system described 
later to determine availability of the limited resource (e.^., training device or lab station) 
and make the appropriate assignment. A second additibri fbr trairiirig j»th selection 
function is the assignment of students tb a majbr segrrieht of basic skill training (e.g., 
30BS Program). This function wbuld require accessing basic skill diagnostic information 
and making the assignments as space becorries available. The system should also be able 
tb assign mbre than one rribdule tb a student^ although there should be a prioriti2ation 
given along with any multiple study assighmehts. 

two other variables that should be used in making differential training^^^th assign- 
ments are aptitude arid previous perfdrrnance data. Although easy to implement on the 
system^ research shbuld_ be conducted to justify the effectiveness of the differem^ 
assignment procedure. Although some research literature indicates that aptitud 
do have a differential effect on learning, dejjending upon the Jnstructional treatment^ 
operational application should be attempted to determine the practical value to Navy 
CMI. Since a training^th could be varied for s«e vera! rea^ 

time" should be created so that time away from a typical course pattern would be 
properly accounted for. 




C bmpiiter Implicatibn - Expansion of the training path selection process woald not 
require significant additional computer processing, although, as defined, a resource 
allbcatibri system would have to be programmed and it must interact with training path 
selectibn. In view of the anticipated improvements iX A^ud^nt flow an^^ 
isquipmeht usage, this fuhctibhai addition could be very cost beneficial. Benefits frbrh the 
differential assighrheht to course material, as a function of aptitudes or previ^^^^^ 
performance^ should be verified through research before system-wide implementation. 
Since the fUhctibh would not be costly to develop during a redesi^gn effort, it could be 
built and tried out on an experimental basis in isolated learning compiexes to obtain 
justifying data. 



Studen^Da ta Input 

Specificat ion. The system should permit the student to enter data intb Jhe system by 
any of the following means: machine-scorable answer sheets, portable electronic test- 
answer input devices, electronic keyboards on microproeessors, or autbmated performance 
test devices. Ideally, input data could be either numerical or alphabetic in nature^ or a 
response defined by a skill performance task (e.g., teletyping skill). It is hot necessary to 
provide all the data entry mechanisms for each learning complex but, rather;^ only those 
data input methods that are needed. Given the state of the art, application of the 
electronic test-answer input device should clearly be implemented on a wide-scale basis 
and keyboard equipped processors should be installed fbr thbse courses with a sizable 
requirement for either constructed response testing or fbr performance tests that could 
be simdlated in a two-dimensional device. j 

Computer Implication . Since costs for all of these input devices could be prohibitive, 
appropriate justification should be provided to school implemehtatioh. Developing a 
standard interface specification fbr the autbmated performance tests (APT) would be 
highly desirable, since APT implementatibh to date has been on a case-by-case basis (e.g., 
TIMES for the RM school and NEWTS fbr the EW school; in both of these cases. 
Interfacing has been problerhatic). If the interface were properly developed nd only 
summary data submitted tb the central Mainframe, there would not be a significant effect 
on the central system. 

Instructbr Data Irlput 

Specif icatibh . Ihstructors should be able to enter data by the same means available 
tb the students. It shbuld be expected that there would be instructor terminals available 
for making specialized data requests. 

Computer Implication . Permitting instructors to enter data on the same devices as 
the students requires only proper coding. If the instructor is making variable data 
requests, the system rnust have a flexible data base to satisfy the requests in a shbrt time 
(a few minutes). Having an easily accessible data base absolutely requires redesign of the 
current system and will probably be expensive. 

lest Scoring/Feed back 

Specification . Test scoring should be expanded tb ihclijde scbrihg of cbhstructed 
response (fill-in) type questions. Data exist_that dbcumeht the increase in Information 
retention by use of valid fill-in questions. Proper analysis shbuld be done to document 
those cases where fill-in questions are necessary and cbUld profitably be implemented. 
The ISD procedure does require constructed response testing for certain kinds of 
objectives. A fiil-in test item prbcedure, using only a limited string of characters (no 
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rhbrc than 20) and ah inexpensive electronic test-answer input device, might be deveioped 
and implemented cost effectively. Other scoring procedures that should be a vaiiabie 
include the scoring of automated performance tests. The co^rnputer system should b 
designed so that the APts are scored as close to the test situation as possi 
costly transmission of response data and resulting RT and DT problems^ One application 
of fill'in testing would involve on-line testing (computer-generated tests) where the 
student receives the test questions directly from the computer. This would require more 
student terminals, since a student uses the terminal for the entire duration of the test. 
Again, this function would have only a limited application. 

Computer Impiication . Scoring of alphabetic fill-in iterris reqijires relatively sophisti- 
cated programming to accommodate misspellings and alternately spelled words that are 
actually correct, the programming would have to be developed. If the testing and 
scoring procedures were conducted on a microprocessor and only surrimary data submitted 
to the central system, there would be no negative effect on the overall system and scoring 
errors and effects of system RT and DT problems would be minimized. Careful analysis 
should be performed to ensure that use of these procedures will be cost effective arid riot 
merely satisfying an ISD requirement. A major issue in scoring iriyolves decidirig where 
the actual response data should go for scoring._ Should it stay at the iriput device or should 
it go all the way to the central Mainframe? There are pros and coris for both apprdaches. 
The use of on-line testing or cornputer-gerierated testing would not affect the Mainframe 
if the tests :wer(e done on adjunct micrbprbcessbrs, but it certainly would iricrease central 
processing requirements if the Mairifrairie were expected to handle that capability. Not 
only would the computer be tied up during testings but the course files would have to be 
expanded to include the actual test question text arid all the appropriate response options. 
Again, this function could be cbrisidered iri the redesign arid structured as a researchable 
area, irripleriieritirig the capability drily after justif yirig data are avaihible. 

Remediatidri/Prescrlptldri 

Specificatidri . Expansion of this function includes providing the capability to 
differentially assigri alternative versions of training materiaj for within module remedia- 
tidri purposes, .deperidirig on aptitude or prior performance data. This function has a low 
pridrity fdr implementation, and cost data are needed to determine whether it contributes 
to training effectiveness or not. 

Computer Implication. Providing the capability to assign different materia,ls would 
be quite easy and cheap from a cornpOter perspective. The high cost area, however, would 
be that of developing alternate media forms for the same learning objective. 

Student Progress Tracking 

Specification . Student progress tracking should be exparided io provide the iristrUctdr 
with up-to-the-minute data regarding student perfbrrriarice^ as bpposed tb data currerit to 
the preceding day. These data could be used with the iriceritive charts, develdped by 
fsJAVPERSRANDCEN for students and instructbrs, tb imprbve student flow through an 
individualized course. Additionally, the data could be used iri Acaderiiic Review Boards, or 
in conjunction with a flagging system that wbuld alert ari iristrUctdr td a studerit who had 
seriously exceeded predicted stijdy tirrie. 

Cbmputer Implicatiori . Prbvidirig up-td-the-minute student progress data is not 
possible with the preserit system without restructuring the data file system. It would be 
easy arid cheap tb prbvide this fUrictidri, if a redesign effort were performed. 



Management Information Reporting 



Specification . The focus for this furictibrL.is to irnproye the retrievability of stored 
data tor all users within the training system. Up to the minute data for active studehts 
should be .available to the instructor or school oh a qUick-respdhse basis (within 10 
minutes). Student perf brmahce summaries should be developed that are cdhcise and easily 
available. Summaries for historical data should be available within a few days without 
special programming. It should be possible to bbtaih group data for CWI system students 
categorized by derribgraphic or aptitude variables. Management should be able to obtain 
surrirhary reports oh the cost effectivehess of individual schools or the expanded CMI 
system as a whole. 

C bm put er I m pi i^atioft . Improvement of data retrievability will require a redesign 
effbrt. Tb have sbme of the desired information available at the school level 9n_a J"_apid- 
respbnse basis may well require storing of the information on^ a cojriputer at the 
geographic site. Providing ma^nagemer^t reports with system evaluation information will 
require adding data elements to the system but is quite feasible during a redesign effort. 

Instructional Program Development 

Specification . Improvement of the instructional development function involves both 
authoring and course validation. An ability to diagnose course coding (authoring) problems 
should be developed on the system as the minimum addition. With the expanded system, 
course coding should be performed on-line, with a series of instructions provided to the 
coder, to ensure that the coding process is performed efficiently and accurately. In terms 
of authoring instructional text, the expanded system might have the provision for IPDCs 
to perform text authorin^^ atJeast within their brgahizatibh. Practically^ it wbuld hbt be 
possible to implement full ISD authoring throughout the expahded system^ giveh the 
constraints of limited tirrie and resources. As ah initial phase of developmeht^ hb 
portion of the computer system should be rriade available for limited cburse aUthbrihg, tb 
develop the kijids of procedures that would surely be irhplemehted ih a h^wly developed 
CBI system. This very limited authoring capability could be part of a research effbrt 
conducted ih a lirriited context (e.g.^ in one school). 

Cburse validatibh shbuld be expanded by making performance data more readily 
available tb cburse develbpers and develbpihg a student assessment procedure. Students 
would respbhd to ah attitude questibhhaire and enter their responses into the system as 
they hbrmally db. Of greater importance is the provision for irnpf'pved course material 
validatibh by mapping fleet performance data with learning objectives through objective- 
cbded test items. This would improve the ability to identify deficient materials that had 
a negative effect out in the fleet. 

Com putec -Implicati on. The establishment of a.ny on-line system, be it coding, text 
authoring, or studen^t testing, will have a signi^^^ effect on the computer system, 
P?r?l^^'%C'y if the function^ is implemented system-wide^ The effect would include a 
greater requirement for computer processing, which might be offset by improved course 
operation due to better course coding. The text authoring would be particularly costly 
but, if implemented, it would permit course designers to develop rriaterials (tests) and 
immediately send them out for tryout. If successful, the tests could be used immediately 
with no delay or additional expense for nrioterials reprbductibh, mailings br insertion. As 
desirable as this function may seem for cburse develbpers, it cbuld be very costly tb 
computer operation. Implerhentihg both the authoring ahd validation capabilities ih ah 
R&D complex of a limit^ scope would jperrhit bbtaihing justifying inforrriatioh with little 
cost and low possibility for major system defect. This functional capability shbuld be 
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included in any initial redesign recquiremeht bh.a limited basis to make sure that it would 
be possible to implement it system-wide in the future. 

Training PrOj^ram Validation 

Specification . This added function should permit the determination of the extent to 
which the course is meeting fleet objectives by relating fleet periormance data back to 
test items and learning objectives. Additionally^ 'this fuhctidri wduld^ermit the assess- 
ment of overall system effectivehess by integrating training and performance data with 
pp>eratibhal training costs arid iristructidrial develdpmerit costs. Much of this information 
is already in existing data systems arid it wduld simply have to be brought together and 
integrated usirig apprdpriate models. This cdmpdrierit would also assist in determining the 
relatidriship betweeri achieveriierit across courses or components of courses. 

CdriipUter Impli^oatidn . This function involves the integration of existin^^ 
structures with riew data elements not presently on the system If a redesign effort were 
coriducted, the necessary files and software could be provided, with reiativeiy small cost 
in processing time. The processing to integrate the deqred information could not be 
easily done on-line and requests would take time to^tisf]^^^ If files were set and the 
program in place, it would then be possible to develop and obtain a standard report with 
no special programming, other than entering certain parameter values such as the timie 
period, schools, or students involved. At present, no resources are available to develop 
this function. 

Resource Allocation 



Specification . A resource allocation system that wduld schedule studerits, instruct- 
ors, and equipment is required for the expanded system. This alldcatidri systeni should 
permit students to be called to use a limited resource such as a trairiirig device arid then 
sent back to the normal course sequence. This system cdlild iriiprdve student flow and 
would make better use of ecquiprnerit. A standard, equipnierit iriveritory report should also 
be avaHabie. It should monitor the .availability of schddl sUpplies/equipmerit arid provide 
flagging functions at appropriate tiriies td persdririel respdrisible for procurement and 
maintenance. 

Computer Implicatidris . The resource alldcatidri syst«*riri adds an additional accounting 
task to the computer but it shduld preserit rid probleri; if constructed during a system 
redesign. It would be necessary fdf this system to Interact with the^raimng^th 
selection system td erisure ari drderly arid coritrolled flow of students. Some additional 
irifdrmatidri reports cduld be provided to inform school personnel about equipment and 
supply availability. 

Iristructiorial Syste m Security 

Spe cification . A security function is retired ta maintain the integrity of tests ^ 
student data, and use of the computer, ^th more foictions handled by the computer 
system on the expanded CM! system, secqrity would be maintained through tighter 
application of computer codes and instructions. Ajnonitoring system should be built irito 
'the computer to identify improper use of the system. This function is like that described 
for the ADP-GP system. 

Computer Irnplication. The addition of codes and computer iristructidris shduld add no 
great load on the computer. Having the computer scramble test item respdrise alterria- 
tives increases processing slightly; however, if the items are already dri the system^ iri the 
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form of on-line tjesting, there would be no increase in computer requirement. With more 
of the school's tests on the computer, there should be at least a perception of greater test 
security; 

Historical Record Keeping 

Specification . Ab with the ADP-GP system^ the expanded CWI systein shbUld have 
irnproved record kee pi rtg capabilities with the data stored at the central site, except fdr 
active students, bataxfor active students should be retrievable from the geographic 
location, the historical dSta- would be accessed through the management information 
reporting function. 

Co rnputer Implication . The major implication for the computer is the requirement 
for greater storage at the Mainframe and an improved data management capability 
(random access to files) to service the jncreased number of types of requests. It would 
not be exp>ected that the historical data could be accessed immediately. Responses to 
requests would require several days to be satisfied but should not require special 
programming. 

eMI SYSTEM UPGRADE AtTERNATlVES AND ESTIMATED CeSTS 

The alternatives for GNET to consider for upgrading the GMI system evolved in part 
from the analysis of user need information. The alternatives include the following: 

1. Perform rriiriimurri sylterri rnadificatioh that will provide, at least temporarily, ah 
immediate resolutibri to the RT arid DT problems. 

2. Perfprrri the riecessary arialysis arid^ if warrarited^ redesigri the CMI system to 
provide brily the currerit set of required CMI f urictibris. 

3. Perforrri the analysis and redesigri tb cbmbirie required CMI functions with the 
desired set bf furictions fbr A DP suppbrted grbup^paced iristr uctibri. 

^. Perform the analysis and redesign the system tb prbvide expanded CMI capabil- 
ities cbmbinisd with ADP support for group-paced instruction. 



An additional option for CNET, although not directly involving the CMI system, is to 
develop and acquire independent ADP systems for supporting group-paced instruction. 
This report section will discuss each^f the CMI upgrade alternatives and present 
preliminary cost estimates. The cost estimates developed by MIISA should only be viewed 
as rough approjdmations. Only after more extensive analysis and refinement of system 
specifications can accurate costs be determined. 

Aiternative i--Irnrnediate Resolution of RT and DT Problems 

Resolution of RT and DT problems through application of rajSid management actions 
will improve system operation, at least on the short-term basis, and is necessary to 
minimize current student backlog conditions. This approach will prbvide continuation of 
existing required CMI functions. Many of the actions that are possible have already been 
implemented during^ the past 2 months arid there has been significant improvement in 
computer RT. To obtain this short-term RT improvement, however, it has been necessary 
tb alter sbme of the computer subsystems and tb. mbdify some school administrative 
practices. While the changes iri schbbl administrative practices aje endurable over the 
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short rurtj a rT1d^c permanent fix needs to be developed, MIISA has already cbrripleted the 
fdildwing ictidnsfe 

L Files have been split to isolate major files used by EVAb of to separate disk 
drivesi ^ 

2. The RSiSAN prbgrarh has been revised to mihirriize and isolate effects of long 
learning guides. 

3. Errors in equipment bperatidri have been cdrrected. 

The total effect of these changes has been to improve RT, but they have not jjinpointed 
the source of the prdblem, CdhsequeritlyK irnprdved system operation may once ^ga in 
deteriorate as sddh as additidrial student load or more courses are added to the system. In 
fact^ the additidh df prdcessing requirements onto the compute^r f^or sqm^ 
data systems may bring back the long RTs that are so frustrating to student and instructor 
alike, the system is saturated. 

Certainly MIISA should and will continue its efforts to isolate the factor or factors 
that are responsible for degraded system performance. For example^ MIISA is revising the 
registration program to increase system efficiency. It has been necessary to add a 
minimal amount of_personnel and equipmerit resources to provide these changes. The 
estimated cost for the purchase of equipment is $33,060 and there is an increase of $330 
on monthly maintenance. A one-time cost for developmental labor is $2ly000y direct 
salary only. 

Alternative 2--RedesiRn System to Provide Only Required CMl Furictibhs 

If analysis reveals that a redesign is necessary tb perrriariehtly resolve RT and DT 
problems, this upgrade alternative will result in a system that would prdvide dhly the 
current set of required CMl functibhs and wbuld be expected to be the lowest cost 
redesign effort possible. The redesign v'buld prbbably require additidhal persdhhel and 
equipment at either the MIISA Mernphis or the MIISA detachrhehts. A major design 
decision that would drastically vary the applicatibri of persbhhel arid equipment is whether 
the evaluation, cbmpbneht bf the cbiriputer system shduld cdritinue td reside in the 
Memphis Mainframe br should be distributed td the cdmputer at the Ideal sites. It needs 
tb be emphasized th;at rrierely_ distributing evaluation to the geographic site does not 
guarantee irriprdved RT and DT. While that step would reduce problems associat;ed with 
Ibhg-distance line trarismissidri errors, it may only serve to dUtribute the RT and DT 
prdblems. phly by perfdrming detailed analysis to pinpoint the e5cact source or sources of 
system prdblems can they accurately be identified. Once ideriified, a deci^^ 
made dri how to correct the problem. Since the source of the jjrqblem js n^^ 
time, and since system redesign could involve either adding to the Mainframe capability 
or distributing evaluation processing to the geographic sites level, it is appropriate tb 
estimate v^hat the redesign and implementation oosts would be ior both alternatives. 
Accordingly, there will be two Alternative 2 design estimates: (1) Alternative 2A — Re- 
design to provide required CMl functions by adding to central site^ and (2) Alternative 
2B--Redesign to provide _ required CMl functions by distributing prbcessing tb the 
geographic sites (detachments). 

Alternative 2A--Required CM! by Upgrading Central Site 

This alternative provides fbr an enduring fix bf RT and DT prbblems. Irriplemeritatidn 
would be accomplished by adding to the central site prbcessing capabilities and wbuld 
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require 1-2 years. Many of the system changes are directed at irhprbvihg the entire 
facility and not just improving CMI operation, since the two can't be Isolated. The cost 
estimates, which are provided in Table 2 assume use of current Honeywell equipment and 
are broken into two categories: (1) items unique to fixing the existing centralized system^ 
dnd (2) Items desirable for any system. 

Only the $164, Q0G cost for items unique to upgrading the centralized system should 
be used in alternattve comF>arison. The $863,000 for desired additions is ah expense that 
could be applied to all alternatives should a redesign effort be necessary. What is hot 
included at all is cost for a second Mainf rame to use as a development computer, should 
that be desired. 

Alternative 2B — Required CMI by Distributing Processing 



This alternative also ensures ah upgrading of the required CMI system operation. 
However^ the sblutibh is achieved by distributing the EVAL program processing to the 
local geographic sites. This alternative assumes the use of Honeywell equiprnent that 
would be prbcurred in about 1 year. Further, it assumes the need for a Honeywell 6620 
processor at each site, since MIIS A believed that the Level 6 minicomputers would not 
have the ability to support the EVAL program and that a dual system, simiiiar to that in 
the present system, would be needed at each site^ ^ipGS w[th an uninterrujDtible power 
supply. This approach recognizes the high cost for each site; hence, it is assumed that 
only NTCs at San Diego and Great Lakes would justify sucji a system. Orlando, Pensacola, 
and Meridian would have to receive support from a remote computer. For each site, 
MIISA estimated total one-time costs of $3,230,000, induding equipment, building 
modifications, and jx)wer supply. A single software development expense exists of 
$90,700. There is a.n additional monthly maintenance and operation cost of $65,400 for 
each site. See Table 2 for details of these estimates. To install this fix just in NTC 
San Diego, Great takes, and NAS Memphis requires a one-time expense of $9,780,700 and 
monthly costs of $196,200; that is without providing support to Orlando, Pensacola, or 
Meridian. 
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Table 2 

Cost E?;tirnates for Alternative 2--Reqaired CMl Functions Only 



Item 



Cost Estimate ($) 



Alternative 2A--Redesign to Required CMI by Upgrading 



Central Site UpgradinR; Uniqqe to Gentraiized Processing 

One-time Costs ; 

Equipment (second EVAL prograni) 
Labor (development) 

Monthly Costs (Maintenance) 

System Upgrade: Desirable Additions ^ 

bne>tjme Costs ; 

Equipment 

Labor (development) 

Monthly Costs (Maintenance) 



150,000 



Total One-Time Costs for Alternative 2A 



16^1,000 



211,000 
652 yTOg 

863,000 

1,000 

1,027,000 



Alternative 2B--Redesigh to Require d CMI by Distributing 



Central Site: One-time Costs 

Equipment Development 
3.5 man-years 



Total One-time 

Remote Site; One-tiriie/Site 

Equipment; One-time 

Building mbdificatibh (1) 
Air conditioning 
Power supply (1 ) 

6620 Processor (256) (2) 
Datanets (2) 
Removable Disks (S) 
Tape Drives W 
1200 LPM Printer 
300 CPM Reader (_1) 
LOO-/. 00 CPM Punch 
CbmmUhicatibhs channel*. (30) 



90,700 
90,700 



80,000 
100,000 
150,000 



Development 
Total Ohe-tirfie/Site 
Remote Site: Monthly 

Maintenance; Monthly 
Contract 

Field Engineer (2) 

Operations; Monthly 

App. Systems Programs (6) 
Site coot dinator { I ) 
Operators (6) 

Total Monthly/Site 



2,900,000 



3,230,000 



32,000 
10,000 



r^,4do 

1,700 
7,300 



65,40 C 



These changes provide clcrrients in cdmmunication system, centralized course coding, 
Recess mbnitbr, second^ tape backup, fribvlrig ARTs to Level 6s, broadcast to terminals, 
and improved diagnostic monitors. 
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Alternative 3--Redesigh Systerh to Provide f^eqUifed CMI FunGtions- Combined ^Uh-ADB 
for Group-Paced Ihstructibh / 



This Upgrade alternative Involyes redesigning^ the CWI system to support current 
r*^quired CWl functions combined with ftDP supported functions for group>-paced instruc- 
tion. To provide a dimension of si^ze to the group-paced portion of the system for costing 
purposes, the following paramet^ers regarding the scope^ of support were selected. The 
selection was done by applying the function specifications from the previous report 
section to a. set of potential school users of the ADP-GP system. A brief summary of 
ADP~GP system scoring specifications is that it will provide the capability for classes of 
students to have 100 item rhuittple-chotce tests scored and summary data provided to the 
classroom level instructors within 15 minutes. Scoring will be performed so that no less 
than 300 tests will be graded per hour at any one school location. Instructor feedback will 
include a list of student SSNs or names, individual test scores and missed objectives, _ahd 
an average class score. Of course, administrative support would also be provided. This 
capability can easily be required now with existing off-the-shelf equipmient. 

The potential schools for group-paced ihstructibh ADP support were identified by 
reviewing the school's quesjiohnaires. Schools selected were those that indicated some 
degree of interest in _ ADP support. This list of schools represents a hypothetical 
application of the ADP system and should not be viewed as a precise specification of 
schools that must have or that demahd_this. support. The schools^ their AOBs, arid their 
geographic locatibhs are preserited_Lh Table 3 tb provide the reader with a perspective 
regarding the pbssible extent tb ADP suppbrt fbr groufvpaced ihstructibh that might be 
necessary br desired. This sarhe list of schools will be used in cbmparihg all upgrade 
alternatives involving grbUp^paced instruction suppbrt. It should be mentioned that some 
bf the schbbls on the list desire group-paced ihstructibh ADP suppbrt, ^IthbUgh they use Zk 
fbrm bf individualized instructibn. These schbbls were included since some schools will 
indeed want tb pursue this apprbach. It is difficult tb predict how well it would work 
bperatibnally. 

Applying ADP-GP Instructional system specifications to the school information 
contained in Table 3, the following estimates of testing requirements are derived. 

1. There should be course file storage and A^'9^?L°n__Pr°^5^^^^8 sites 
presently supported by MIISA and at four sites not supported by MIISA (Guifport, 
Meridian, Port Hueneme, and Mare Island). 

2. There will be ART programs at three sites, average AOB of ]7BQ heeding only 1 
test/week in batches of 599. 

3. There will be apprentice training at three sites, average AOB 763, requiring 
testing twice/ week in batches of 100. 

4. There will be a total of 2^8. "A*' schools having approximately a total AOB of 
6362, in an estimated 127 classes of 50 students each. 

Each class requires testing 1/day. 

As with Alternative 2, the apprbach tb implemehtihg this design concept can be to 
add to existing Mainframe capability at Memphis fbr all evaluatibn, br to distribute 
evaluatibh prbcessihg tb the gebgraphic site level. Accbrdingly, Alternative 3 is 
estimated fbr bbth cbnditibns. These conditions are Alternative 3A--Redesign to Provide 
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Table 3 



A Preliminary List of eandidate Schools for 
Group-Paced A DP Support 



Location 



School' 



AOB' 



Great Lakes, IL 



San Diego, GA 



Memphis, TN 



Orlando, FL 



Gulfport, FL 
Meridian, MI 



Port Hueneme, CA' 
Mare Island, CA 



Apprentice Training 
Fire Control Technician 

Gunners Mate 

Electronics Technician 

Daily bri Board 

Academic Rernediation/Week 

Apprentice Training 
Data_Prbcessor 
Hull Technician 
Messman School 

Daily on Board 

Academic Remediation/Week 

Avionics Electrician 
Tradevrhan. _ 
Advanced First Term Avionics 
Air CbhtroUer 
Aviation Ordriariceman 
Aviation Structural Mechanics 

Daily on board 

Apprentice Training 
Quartermaster 

Daily on Board 

Academic Remediation/Week 

Builders School 

Yeoman 

Storel<eeper 
disbursing eierk 
Personnelman 
Aviation Storekeeper 
Administration 

Daily on Board 

Seven Small ''A" Schools Total 
Data Systems Technician 
Total (Not including ART) 
Total (ART/Week) 



891 
*25 
.350 
1600 

3266 

1700 

666 

125 
160 
»0D 

1351 

1700 

*50 
28 
366 
294 
275 
128 

1535 

732 
120 

852 

1700 

120 

2*5 
115 
50 
120 
100 

685 
370 
352 
8531 
5100 



^Schools were included if they indicated the madest interest in A DP support arid did not 
express any defiriitive or formal statement regarding acceptance of A DP. - • 

^AdBs were obtained from a variety of sources and should be viewed only as 
ART AOBs were determined by dividing total ART load by the three locations. 



estimates. 
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Required CMI Combihed wijh ADPrGP by Adding to Central Site Capability, and 
Alternative 3B--RedesJgh to Provide Required CMI Corn bjhed with ADP-GP by bistribut- 
ihR Prbcessihg to the Geographic Site. Again, the cost estirnates are only preliminary and 
should be considered as an indication of the order of cost magnitude for each alternative. 



Alternative 3A--Rede-sigfi-£ or Centralized CMI with ADP -^^GE 

This alternative involves developing and adding the ADP-GP system to the existing 
centralized CM! system. There ar? four major categories of costs to be considered: (1) 
incremental costs at the central sit'='. (2) incremental costs at local sites already 
supported by CMI, (3) costs at sites no c supported^ by CMI, an^ (4) costs per terminal. 
Table 4 provides detail for each cost 3rea. Total cos^s are computed for the four CMI 
supported sites (SanJDiego, Memphis, Great taJ<es, and Orlando) arid the four non-GMl 
supported site^s (Gulf port. Meridian, Pq Hueneme^and Mare island) and are presented in 
Table 5. Each termirial will support approxtniately 1908 students^ assuming some 
schedulirig among schools for test sconrig, thereby requiring 13 terminals for the eight 
sites as presented in Table 3. 

Alternative 3B--RedesiRn for Bistributed CM! with ADP-GP 

This alternative wiH provide the ADP-GP system in conjurictiori with a distributed 
CMI systeni. Again, the four major cost categories include: (1) costs at the central site^ 
(2) costs of local sites already supported by CMI, (3) costs of local sites riot supported by 
CMI, and (4) terminal costs. The same parameters regardirig the scope of ADP-GP 
support as were applied to alternative 3A apply to this alterriative. The develbprrierit 
costs for non-CMI sites assume that devejbpmerit for CMI locations has already been done. 
Table 5 also presents the sijmmary costs for this alternative. 



44 



Table^f 

. .1 . . ._ . 

Detailed Cost Breakdown for Alterhatiye 3A~Redesigri 
for Centralized CMI with ADP-GP 



Item Cost 



Central Site: On e-time 
Equipment 

Dual 66/80 (1) ' 300,000 

128k Core d) 105,000 

Disk Drives W 132,000 

CRT Terminals 2^,000 

Tape Drive (1) 32,000 

DatanetFEPU) 200^^00 

793,000 

Development 

5 man- years 1^3,200 



Total One- time Central Site 936,260 
Central Site; Morithiy 

Maintenance Operation 8,000 



Total Monthly 8,000 
Remote Site^Ex i stingy CMI) 
Equipment 

Level 6 (*3) (approximately) 60,000 

Develbprheht , 

Total One-time Site (approximately) 60,000 

Rerrrbte Site (Existing CMI ); Monthly 

Maintenance Ij^OP 

tine (approximately) 800 

Operation 

Operator (1) l»20d 

Site Coordinator (1) 1,600 



Total Monthly Site (approximateiy) 5,000 

— — < 




Table 5 



Cost Estimates for Alternative 3 





One-time 


Maintenance 


Cost Item 


Cost ($) 


Monthly ($) 


Alternative 3A--Centralized CM with ADP-GP 


Central Site ADP-GP Costs 






Equipment 


793,000 


8,000 


Development (5 rriah-years) 


1^3,200 




Remote Site (already supporting CMI) x4 






EqUipirierit--$60,000 x ^ 


lib , boo 


20 , dob 


Rernote Site (not supporting CWl) x ^ 






Equipment--$73,000 x 4 


292,000 


40,000 


Terminal Cost 






Scantron 


3^000 




Printer/ Keyboard 


5,000 




Link and Modems 


500 


200 


Terminal Total Cost Each 


8,500 


xi3 


X i3 = Total Terminal Cost 


110,500 


2,600 


Total Cost for ADP-GP System 


1^ 578 j 700^ 


70,600 


Central Site-Upgrade-f rom 






Alternative 2A 


1,027,000 


2,400 


Total System Cost 


2,605,700 


73,000 


Alternative 3B— Distributed CMI with ADP-GP 



Central Site 

Development Costs 
($90,000 + $143,200) 

Remote Site (already supporting CMI) 

From Alternative 2B to obtain 

distributed processing 
For ADP-GP Equipment 

Total per Site 

X 3 Full Sites 

Remote Site ADP-GP (no CM! support) 

Equipment 

Development (2.5 man- years) 

Total Costs per Site 

X 5 (includes Orlando) 

Terminal Costs (13) 

Total from Alternative 3 A 

Total Systeih Cost- -Alternative 
3B 



233,200 



3,230,000 
63,000 

3,293,000 

9,879,000 



245,000 
67,iOQ 

312,100 

1,560,500 

.110,500 
U, 783, 200 



65,^00 
5,300 

70,700 

212,100 



20,000 
20,000 
100,000 

2,600 
314,700 



^The cost of $1,578,700 will provide' only the ADP-GP system where designated in Table 
3. 
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Atternative ^--Expanded CMI Combined with A DP- 



If a system redesign is necessary, then CNET should consi^der adding ca pa biii ties to 
the current CMI set of functions and combining those expanded capabilities with the 
functions necessary to provide grquj>-j5aced instruction with ABP supp6ri._ Because of the 
projected growth in computer requirements to^satisfy the expanded CMI functions, it was 
determined that both addition to the Mainframe capabHity 

to the geographic site would be necessary for successful system operation. Costing of this 
alternative was difficult due, in part, to the ambiguity of the requirement^ The cost 
figure used was determined by using the high estimate for alternative 3B--Redesigh to 
required distributed CMI with ADP-GP and adding increments for certain features. This 
estimate is shown in Table 6. 



Table 6 

Cost Estimates for Alternative ^--Expanded CMI 



Item 


Cost ($) 


Alternative 3B- -Total Cost 


11,783,900 


Resource allocation 


55,000 


Simple constructed response scoring 


30,000 


Coding of iterris to permit use of training 
validation program 


lif5,000 


Maintenance of detailed learning rate data 


111,000 


CRT terminals for on-line testing 
200 X $1,000 each 


200,000 


Total 


12,32Zf,900 



The expenditure for Alternative ^ would provide a thoroughly enhanced CMI syst 
with ABP support for group-paced instruction and would use distributed processing to 
ensure system reliability, the expanded capabilities would include: a resource allocation 
system, simple constructed response test scoring, lirnited on-line test^ 
program validation systerri that would enable relating fleet performance data with 
detailed school perforrhance data. It should be emphasized that these cost figUres are 
only tentative estirriates for this alternative. 

Ah Independent ADP- GP System 

Although the task group was charged with exploring alternatives foi_up:gradihg the 
CMI systerh, an additional, option open to CNET is adding independent ADP-GP systems. 
While we have not systematically, explored options for independent systems, some rough 
approximations of costs were developed by MIISA. One independent system was assumed 
to require evaluation processing at one geographic jocation, with fbUr remote schools: 
three schools operating four classes of 30_studehts each, with each student requiring 1 
test/day, and one school operating with 700^ students in classes of 100, each student 
requiring 2 tests/week. Data ^jitry is to be assumed to be by rnachine-scorable answer 
sheets arid all tests are to be JOO items. 
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Cos is for this dption amount to the estimated cos^tjnAlterriat brie rehrible 

site ADP-GP system with no CMI support, and two terminal cbrifiguratibns: 

Remote Site Equipment $2*5,000 
beveiopment 67^100 
Two Terminals (3 $8,500 each 17,000 

Total/Site $329,100 

There would be monthly maintenance costs associated with this option even thbiigh 
individual school personnel would operate the terminals. Sijice each terminal configura- 
tion is estimated to be $8,500, includinj^he jjrqces^^^ and scoring device, that 
figure can be used when varying the number of terminals (i.e., students), suppbrted at 
each school. This option is computed with a termi for each 1000 grbup-paced students. 
The physical Iqcation^ of the schools might warrant additibnal terminals tb^ save time 
transporting answer sheets from one buBding to anbther. Once the basic site irivestrrierit 
has been made, the addition of terminals would be a minimal addition tb the basic cost. 

Comparison of Estimated Costs for CMI System Upgrade Alternatives 

Using the infbrmatibn from the pre yi bus repbrt sections^ costs were estimated by 
MHSA staff for each of the six upgrade alternatives arid the independent ADP^GP system. 
Total cost estimates for the six alternatives are presented in Table 7. 

Table 7 

Estimated Net Costs for Each of the'CMI System 
Upgrade Alternatives 



EKLC 



Estimated 

Alternative Cost ($) 



i- -Short-term fix of CMI 




5t, 


000 


2A--Redesign to required CMI at Cenral Site 


1 


,027, 


000 


2B — Redesign tb Required CMI by distributing 


9 


,780, 


700 


3A--Redesign to required CMl combined with ABP-GP- 
Central Site 


2 


,605, 


700 


3B--Redesigh to required CMI combined with ADP-GP- 
Distribute 


il 


,783, 


200 


4 — Redesign tb expanded CMI combined with ADP-GP 


12 




900 


Independent ADP-GP system for one remote site 




329, 


100 



The changes MIISA has been implementing to improve the CN« system ha ve^ 
cbst bptibhs primarily irivdlvirig software and procedural changes^ This approach has 
product sigriificarit short-term (Alternative 1) improvement in RT and DT^ with minimal 
cbst ($5^^000). A periTlanent solution that will ensure good system response as m 
students arid courses are added is considerably more expensive, as can be seen by the 
estimates for all other options. The reader ishould recognize that all of these upgrade 
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alterhatiyes will prbyide improvem to all the data processing reqaireme^^^^^ 

CMU resident in this MIISA Memphis computer system. WM^ are only 

apprbximatibhSj they do provide "ballpark" figures for comparing alternative costs. 

A notable factor is seen in comparing the estimates for the upgrade alternatives. 
That factor is that Alternative 3A--Redesign to provide centralized BMl with ADP-GP 
support, gives the maxjmum amount of system improvement with the lowest cost ($2.6 
million). This is due to the fact that, by retaining centralization, it is not necessary to 
prepare and operate new sites for major computer systems. The redesign work that would 
accompany Alternative 3A should ensure proper system operation even as the CMI or 
ADP-GP systems are expanded. 

Another point that is evident from Table 7 is that, even though costs for ah 
expanded-capabiiities CMI system (Alternative^) are high, they are not that much higher 
than the costs for the alternatives (2B and 3B), which provide distributed processing for 
current CMI functions. The implication of that should be clear--ahy rriajbr rribve to 
upgrade the system by pursuing distributed prbcessihg should also involve expanding the 
GMI system's capabilities. 

From consideration of Alternatives 2 thrbugh it is clear that a major upgrading of 
the system to ensure cbhtihued good system bperatibh is hbt a Ibw-cbst effort. If CNET 
cannot make the investment f or the system upgrade^ the CMI systerri can be operated with 
its present student and course load. Additibhal ihstructibhal support could be provided by 
independent ADP-GP systems. While the independent ADP-GP system probably would 
have a lower cost (study of independent systems was beyond the scope of this effort); the 
independent system wbUld hbt permit as extensive student data retention or use. On the 
other hand, the independent systems would provide some ADP support for a breeder range 
of schools at a relatively lower cost. To study the full range of possibilities, CNET should 
investigate the acquisition of very low-cost independent systems as a means of providing 
the ADP suppbrt for grbup^paced instruction. 

RECbMMjENbftTldNS 

During the course of this effort, the task group perceived several management 
actions that CNET should consider. These actions are contained in the following 
recommendations: 

1. Continue the short-terrn upgrade approach to resolving RT and DT deficiencies^ 
The system upgrade should include modifications that would provide no more than 5 
percent DT at the learning center cluster, even when the host computer is down. The 
upgrade effort should add ADP support for group-paced instruction. 

2. As part of the upgrade effort, task MIISA to perfbrm the necessary analysis to 
identify and isolate precise causes of system malfunction as well as prbvide reliable CMI 
and group-paced instructional support and develop a cbrriputer_5ystem that will provide 
reliable CMI and the ADP support of group j3aced ihstructibh. Only thrbugh this analysis 
can the proper CMI system be develbped- to permanently eliminate RT and DT problems 
and permit controlled growth of the ihstructibhal systerr-. 

3. Establish a CNET grbup tb cbbrdinate all CNET computer programs that effect 
data. file structures and systerri bperatibh on the CMI computer. This group would mclude 
staff responsible for MIL PERSIS^ NITRAS, Feedback Appraisal System, and all other data 
systems expected tb use the MIISA system. (A form of this task has already been assigned 
to TAEG by CNET.) 
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Ensure that TPCs are knowledgeable ahKDut routine CM their courses 

by providing appropriate irifbrrnatiori through the existing CNTECHTRA system rriariager's 
office. 
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APPENDIX A 
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INFORMATION SOURCES 



The rbllpNvihg is a 
rheetirig^ 28-29 October 

Name 

Nick Van Mat re 
P. 3. Scott 
John H. Pinning 
Phyiiis Salop 
Gbrdbri A, Crawford 
Charles E, btndahi 
jsjahcy Perry 
Gene Hooprich 

Sallie bowies 
Ed Scheye 
G. H. Holmes 
Bill Ottehdbrfer 
Bart Thompson 
George T. Buzinki 

a. Ware 

M. A. Bel to 

3ce Haslett 
Bob Milton 

Pat Lee 



list of participants in the first CMI System Upgrade Task Group 
1980, at Pensacola, Florida: 



(Branch Office 
Memphis) 

CNET 

CNET 

CNET 
CNET 

CNET 



N^22* 

N-94 

00A2 
N-72 

N-.221 



OrRanization 


Code 


Phone (Autovori) 


NPRDG 


1* 




CNTT 


N-8 




CNET 


N-91 




GNTT 


N-82iH 


966-5375 


CNTT 


N-824 


yob-jj/ J 


NTEC 


N-7^ 


■7Q 1 fitiQ 1 

/y l-'+'+yi. 


CNET 


622 


922-3356 


CNET Liaison 021 
from NAVPERSRANDCEN 


922-4582 


CNET 


615 


922-4545 


CNET 


N-915 


922-4201 


CNTT 


016 


966-5148 


CNET 


N-72 2 


922-4101 


CNET 


N-93 


922-4201 


WIISA 


34 


966-5726 



922-4402 

922-4201 

922-4447 
922-4161 

922-4402 



CMI UPGRADE QUEStlbNNAIRE RECIPIENTS 

Questionnaires were sent to all Class "A" courses (including AP courses) taughi at 
Naval Technical T. dining^ Command (NATECHTRAeOM) activities under GNTECHTRA 
curriculum control. Questiqn^naires were also sent to ail Training Program Coordinators 
who are responsible for at least one course meeting these qualifications. 

Questionnaires were also sent to the following brgahizatioris: 

eNTiCHTRA, Code 016 
CNTSeHTRA, Code 017 
CNTECHTRA, Code N-S24 
MIISA Branch Office^JVlempbis 

Naval Education and Training Support Center Pacific, Code N=l 

Naval Education arid Trairiirig Program Development Center, Great Lakes 



ERIC 



A-l 



53 



PERSONAL CONTACTS 

The foilovvtng personnel have [»rticipated directly in discussions related to the 
improvement of the Navy GMl systerh: 

Navy Personnel Research and Develdpmeht Center 

Dr. N. Van Matre 
/i Dr. K. 3bhhsbh 

Chief of Naval Education and Training 

Code N-9 

CAPT R- McKay 
Mr. 3. Hassen 
Mr. E. Scheye 
Mr. H. Krukee 
Mr. B. Thompson 

Code N-7 

Mr. W. Ottendbrfer 
Code N-5 

Dr. W. Scahlahd 
CodeOl3 

CDR_S._Kaiser 

Dr. W. Gager ; 
Chief of Naval Techntcai Training 

CodeN-8 

CAPT P. Scott 
Mr. G. Crawford 
Ms. P. Salop 
CAPT H. Svoboda 
Mr. Wilson Thomas 

Code 016 

CDR G. Holmes 
Mr. E. Evans 
Ms. b. kaHvoda 
Ms. M. Naaiteas 
Ms. V. Reed 
tT S. Booth 

Code 017 

Dr. N. Kerr 
Ms. L. Graharri 

Naval Educatibh and Training Support Center Pacific, Code N-1 

Mr. _H, Stevenson 
Dr. R. Kennedy 
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Mr. C. Taylor 
\is. L. tergurn 
Ms; D. Stark- 
Ms. R. Cobb 
Mr. G. Matlock 

Naval Education and Training Program Development Center, Great Lakes 

Dr. J. Carey 
Mr. C. Ross 
Ms. K. Zeola 
Mr. M. Duermyer 
Mr. W. Nordbrock 
Mr. R. Bbhatch 



MahaRerrieht Inforrriatibh and Ihstructibhal Systems Activity, San Diego Detachment 

Mr. C. Plough 

Service School Command, San Diego 

Mr. E. Trapp 
Mr. G. Griffin 

Memphis Information and Instruction Systems Activity, Branch Office, Memphis 

Mr. C. Tilly 
Mr. G. Buzinki 
Mr. C. Weaver 
Mr. t. Sneil 
Mr. F. Arial 
Mr. W. Kisner 
Mr, K. Kuhn 

Naval Weapon Center, China Lake 

Mr. H. Hammerdihger 

Naval Training Eqijiprheht Center 



Dr. C Lihdahl 
Mr. A. Laihg 

Training Analysis and Evaluation Group 

Dr. G. Micheli 
Mr. M. Middleton 
Mr. G. Hall 
Mr. C. Morris 
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CMI SYSTEM UPGRADE REQUIREMENT ANALYSB QUESTIONNAIRE 
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CMI SYSTEM UPGRADE REQUIREMENT ANAtYSB QUESTIONNAIRE 
For: CNTT CMI Schools 

The Navy Personnel Research and Developmeht Center, CNET, CNTT, and their 
functional commands have established a Task Group to upgrade the CMI system. As an 
initial step, jt is necessary to ddcumeht deficiencies of the present system, planned 
expansions, and desired capabilities in order to arrive at the fuhctibhal requirements for 
an improved CMI system. Your assistance in completing these questions as part of this 
Task Group is vital in this requirements analysis process. Please answer these questions 
as compietely as possible. Your answers will be compiled with answers frorh others in the 
CMI community and foj-mulated into a functional requirements ddcumeht for review and 
revision by the CMI Upgrade Task Group. 

This questionnaire is to be filled^ course being taught by means of CML 

The responses should reflect opinions of course management, support persdrihel^ and 
instructors, so please elicit comments from a wjde range of personnel before consolidating 
them into a single set of responses, tist your answers on separate paper. 

1. Indicate name and location of course, CIN, and CbP(s). 

2. List deficiencies or_ limitations in the current CMI system (e.g., frequent 
downtime, slow response^ difficulty in course coding, or limitations on course design). 
Rate each deficiency as 1. Very Serious, 2. Moderately Serious, or 3. Minor. 

3. List any new functions or capabilities that should be considered for the CMI 
system, including those that are already planned but not yet implemented. Indicate why 
each of the functions are needed. 

^- Most CMI courses are designed in accordance with a set of guidelines, but some 

of those guidelines are based on limited infbrmatibri. Indicate areas in which you think 
the CMI guidelines mi^ht^b^ improved if optimal procedures (e.g., different techniques for 
instructor intervention, for testing, or for providing remediatibh) could be determined. 

5. Rate each of the f ollowing^ standard CMI reports in terms of its utility to (a) the 
instructor in the iaboratory or learning ^ c and (b) course or school personnel 

(administrative or managerial) outs[de the learning center. Use ratings of 1. Very Useful, 
2. Moderately Useful, or 3. Not Useful. 

UtiU ty Ratings 

Learning 
_ Center 

Report Personnel 

StudenJ: Learning Guides 
Daily CMI Student Progress Reports 
Deficient Progress Report 
Accelerated Progress Report 
Trend Arialysis Report 
Extra Study Report 
Learning Center Status Report 
CDP Distribution Report 
School Rosters 
Graduation/ Drop Rosters 
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Administrative 
Managerial 
Personnel 



Utility Ratings 

Learning Administrative 
Center Managerial 
Report Personne J^ Personnel 

Projected CMI Graduation Report 
Student Performance Summary 
Monthly Student Flow Report 
Module Distribution Report 
Response History Print 
Student Status Matrix 
Item Analysis Report _ . 

Module Performance Analysis Report 

h. Have you had problems in receiving the above reports in a timely manner? 
Explain. 

7. Do you ever heed information from the CMI system other than that pr^ 

the standard reports? If so, give examples. Have requests for additional information been 
satisfied, and if so, how long, dh the average, has it taken? 

8. How long should various kinds (specify) of CMI data be retained in order to meet 
your heeds? 

^..y . . 

Answers to the following questions may vary from one CDP to another. If so, provide 
separate sets of answers, indicating for each the CDP or CDPs to which they pertain. 

9. What is the maximum AOB arider instruction? 

10. How many new students, on the average, can you handle each week? 

U. How many tests of any kind (module, iesson, comprehensive) does the average 
student submit to the scanner each day (estimate to one decimal pointy. e.g. ^ 2.1)? 

12, What IS the average pay grade of all students (estimate to one decimal point, 
e.g., E-3.2)? 

13. What is the average pay grade of all instructors (estimate to one decimal point, 
e.g., E-5.8)? 
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CNTT CMI SYSTEM MANAGEMENT (N-8): AREAS REQUiRH^G 
EMPIRICAL VERIFICATION 



CNTT CMI SYSTEM MANAGEMENT (N-8): AREAS REQUIRING EMPIRieAt VERIFIGATION 

Empirical (evaluations and/or literature searches with docamentatibri of findings are 
heeded in the following areas: 

a. Criteria for selection of delivery system: 

(1) What factors determine whether a couise should be individualized? (e.g., 
irnpxact on retention and course length). 

(2) What factors determine whether a course should be developed for cornputer 
management? 

(3) What prbpbrtioh of the tests within a course must bns computer evaluated in 
order to make CMI cost effective? (e.g., what do the assignment, tracking, 
and mahagemerits capabilities alone do f or a course in terms of cost savings, 
given the fact that certain administrative and development requirements 
will exist). 

(2f) What are sbrhe workable models for determining cost effectiveness? How 
do develbprrierit costs enter? 

b. Guidelines for determining optimum learning center de^sign arKi o{^ 

respect to such factors as: (1) arrangenient of carreis equipment, 
computer equipment, (2) integrated ve^rsi^s separate^ (3) centralized 

testing centers, (4) student/instructor ratjos in learning centers, laboratories, 
testing centers, (3) length of training day and number of training days per week. 
What is the impact on retention, contact hours and calendar days to course 
completion? 

c. Identification of instructi^qnai design characteristics unique to computer- manag- 
ed courses. Procedures for handling all design aspects that impact on effective- 
ness and efficiency including such facts as: 

(1) Design of total instructional package for a CMI module with respect to 
initial instructional material and remedial loops. 

(a) What is the optimum prbpbrtioh bf the target population that should be 
handled by the initial rhaterial arid what prbpdrtibn should he handled by 
cbmputer-prescribed rerriedial loops? 

(b) What factbrs influerice desigri with respect to proportion that should be 
handled byiriitial rhaterial? 

(c) Shbuld CMI material be a lean straight path with practice and remedia- 
tibri haridled by computer prescription or should CMI jnstructionaj 
inaterial iricbrpbrate internal branching, significant practice, etc., with 
tirrie gairiiad by better students assimilating faster? 

(dj What is the impact of various types of instructional package desigri bh 
retention and time to criterion? 

(e) Does number of rernedials required to criterion affect retention? 
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(f ) Does time to criterion effect retention? 



(g) Does time at which remedial test is administered affect retention? 

(h) What is the optimum size of segments of ihstructidn prior to testing? 
What factors influence? What is the relation of size to retention arid to 
coarse length? 

(2) Design of computer prescribed remedial loops: 

(a) What conditions justify assigrimerit back to drigirial material as remedi- 
ation? 

(b) How often should reassigrimerit of original material occur? 

(c) What factors determine iricorporation of additional instrucrional 
resources as remediatiori? What Type? What conditions warrant such 
assigrirrierit? How frequently? 

(d) When should instructors intervene in the learning process? What kind of 
intervention should take place? 

(e) Should remedial testing of enabljng objectives be handled the same as 
remedial testing of terminal objectives? 

(f) Under what circumstances if any, is remediation without retesting 
justified? (e.g., does faiiure to meet criterion on a minor enabling 
objective require a retest?) 

(g) When should remedial tests be administered? Is there any difference iri 
assigning all 1st remedial within a rribdule at once as opposed to 
requiring remediation be completed, on brie problem area prior to 
assigning remediatiori on another area? 

(h) When should questions be recycled? 

(i) Does respdriding accurately brily_brie tiriie to brie question per objective 
actually equate tb rriastery? Dbes respbridirig to brie remedial test 
immediately after remedial iristructibri equate tb rriastery? 

(j) Shbuld cbmps be required? Kind? Wheri administered? Relation to 
reteritibri? Dbes rerriediatibri bri cbrrips impact long term retention? 

Structure bf assigrimerit, evaluatibri and feedback data provided to student 
ccriiputer-gerierated learriirig guide (e.g., format, content, prediction and pro- 
gress irif brrriatibri). 

Gradirig arid/or ranking in criterion referenced cours^es. What should grade 
reflect? What factors should influence? Does requirement of grade or rank 
impact student performance? 

Validation of course enabling o^bjectives. Can^ so^ and terminals 

still be met? Can some be added (such as certajn theory, or background type 
enablers) t^^^ learning and either shorten time to criterion and/or 

increase retention? 
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g. iiVii contingency plans, type Bnd phasing in and odt of comiDdter mariagerrierit. 

h. Procedures for managing and operating 6Mf schools; 
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APPENDIX D 

CNT7 016: SUGGESTED ADP SUPPORT REQUIREMENTS FOR 
LOCK StEE> INSTRUCTION 



cntt 016: suggested adp support requirements for 
lcx:k step instruction 



SYSTEM UTILIZATION PARAMETER 



The ADP system will require the ability to manage students in courses with the 
following characteristics. 

1. Courses taught in multiple locations. 

2. Courses with lengths of one week to in excess of three months. 

3. Classes varying in size from 12 to 6(3 students. 

4. Courses with peak loadings as high as 17,000 students; 

5. Courses with classrooms and laboratories located in different buildings at a 
single training site. 

6. Courses with laboratories that contain multiple stations for the same purpose or 
labs containing as many as 218 unique lab stations. 

7. Test schedules that range from fixed periods each day to as many as 12 tests 
occuring randomly throughout the instructional day. 

8. Test design for specific purposes such as erid-bf-lessbh or remediation and test 
banks which can be used either as ehd-of-lessbh or as remediatibh tests. 

9. Courses that cohtairi less than 10 percent cognitive perfbrmahce testing and 
courses that contain less than 10 percent cognitive testing. 

10. Courses that require three or four shifts each day. 

11. Courses in which the student moves frbrh rbbm to room for ihstructibn, lab 
practice^ performance or cognitive testing. 

12. Course with multiple instructional tracks (sequences). 

13. Courses that may be all groui>-paced, all self-paced or combinations of the two 
management strategies. 

14. Courses where remediation is provided d 

hours. The extra study periods may be either before or after the normal shift houis. 

15. Performance testing may be instructor evaluated or direct evaluation via an 
interface between operational eqaiprheht, trainers, or sirnulators and the ADP system. 

16. Cognitive tests may be either recognition or recall questions including i Mmerical 
computation where the process is as important as the correct answer. 

17. Instructors staff may change frequently allowing for little or no preliminary 
instruction oh how to interface with the ADP system. 
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18. babbratbry situations may create extreme heat or generate significant amounts 
of dost or other airborne particles, 

19. Tests may need to evaluated in 20 minutes or less to allow for in class 
remediation usually scheduled during the next instructional period. 



K Registration: 

a. Enroll student in the system 

b. Identify assignments (track, multiple tracks) 

c. Site identUication 

d. Set-back/advance student 

e. Suspend student's file for time or grading purposes 

f. Change student data entered incorrectly 

g. Administratively drop student from course 

h. Provide roster for instruction 

2. Progress Reports 

a. After each evaluation, list of students in various forms: 

(1) alpha students who passed 

(2) aljjha students who failed 

(3) SSN students who passed 
in) SSN students who failed 

(5) alpha students who missed test for authorized/unauthorized re^^^^^ 

(6) SSN students who missed test for authorized/unauthorized reasons 

b. Time heeded to complete assignments, by individual module or entire course on 
dlass basis^ individual student basis, or overall average. 

3. Night Study (MS) Reports 

a. alpha list of students assigned to NS 

b. SSN list of students assigned cO NS . . 

c. Areas requiring remedial instruction (list of most missed down to least missed 
items, e.g.) 

^. Item analysis (by class, overall) for each test and overall Module Performance 
Analysis Report (by class, overall). 

5. Performance Report- -Overall student perforrnance in school, i.e.^ graduation, attri- 
tion data by various categories (CDP, tirne, mental category, etc.) 

6. Attrition Report- -Number of arid other pertinent data bri students dropped in a 
particular timeframe. 

7. Advance/Set-back Report 



a. List of students set-back/advanced 

b. Overall statistics, percentages of set-backs/advances by class 



SYSTE M FUN 
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8; Response History (responses and tirriej- -Individual rhbdule or entire course. 
9. Student IVlatrix--Eriables students track to be identified, modified, etc. 

10. Student 5tatus Repbrt---Perfbrmarice of individual student at any giyeh tirtie ciDuld 
include grades (if not pass/fail) made on each exam; any exams missed (if prerequisites 
are not set from. brie module to the riext); riumber of NSs*^ etc, 

11. Class Rosters (alpha/SSN/rate) 

12. School Rosters (alpha/SSN/rate) 
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DESGRIPTION OF GENERAL INSTRUCTIONAL SYSTEM FUNCTIONS 



Fanction 



Description 



Delivery of Instruction ; 



The total set of activities involved in delivering 
instruction including providing subject matter by 
means of lectures, discussions, reading material, or 
student iriteractidh with teaching rriachihes or simu- 
lators. 



Instructional Mahaj^emeht ; 



Student Registration 



Training Path Selection 



Data_Input: 
Student 
Instructor 



Test Scoring/Feedback 



Remediation/Prescription 



The total set of activities that a schoblhbuse must 
perform to guide a student through ah instructional 
sequence ainS result in successful attainment of the 
learning objectives.: 

Student registration provides for the ehrdllihg of a 
student into a school and course. It involves record- 
ing available student aptitude and dertldgraphic 
data. In the case of Navy individualized courses, 
registration also involves the specific assignment of 
the student to a particular study carrel. 

Ji?^iy[!E _f??6 JS^Ll^tion involves the selection and 
assignment to ^ student of ^ particular sequence of 
instructioiiai modules or lessons within A_?P^r??* 
While this activity can be automated in computer- 
based systems, the capability must exist for an 
instrtjrior to override a pattern, module, or lesson 
assjgoiucnt, assuming appropriate admiriistfative 
approval. 

Data input are those activities that involve a stu- 
dePit or instructolr subrriittihg data into the instruc- 
tibhai system. Inputs can involve answers to test 
questions or requests for feedback from the system 
regarding student perf orrriahce, Jbcatioh in the 
course^ or subsequent assighrrient. The input method 
depends upon the school arid can vary from pencil 
and paper to the use of interactive computer ter- 
minals. 

Test scoring consists of those activities related to 
evaluating the accuracy of student responses to 
knowledge or performance tests, questions, dr 
exercises* The feedback function provides the stu- 
dent with knowledge ofiTresults of the test or per- 
formance evaluations. 

The remediation/prescription function proyidjes the 
studer^t with expiidt guidance regarding the study 
actiyitie5: the student should engage in to overcome 
a within-module 9^ Je^on-l^earninj objective defici- 
ency, identified through the test scoring function. 



Description 



Remediation/Prescription, Cont. 



Student Progress Tracking 



Management Information 
vrtihg to: 
Student 
Lhstruc' or 
School 
Mahag^meht 
Course Developers 



Instructional Sufipo^ i 



The indicated remedial activity is delivered through 
the test feedback function and includes a specific 
reference to the next test activity the student 
should perform following completion of assigned 
study activities. 

Student progress tracking provides the capability to 
monitor the student's progress through a sequential 
course of instruction in terms of achievement arid 
time. In addition to tracking progress, this furictibh 
provides the capability of predicting student course 
completibri time and forecasting graduation. 

This furictibh cbrisists of the repkjrtirig of studerit 
progress arid achieveriierit inf olmatiori to each level 
in the iristructibrial system. This iricludes develops 
irig arid seridirig reports of individual progress direct- 
ly back to the studerit or instructor. It particularly 
iricludes summary reporting of information about a 
group of students to the instructor, school, or higher 
mariagement levels or to the instructional program 
developers. 

Instructional su]5port functions are those activities 

mus^t perform to ensure con^tinued operation of the 

^9^_°9^> PPl they don't directly involve r^^ 

of the instructional activities of the schoolhoase. 



Instructioral^rogram 
Development: 
• Authoring 
Validation 



Instructional program development includes those 
activities necessary for the design and authoring of 
instructional materials, and those activities associ- 
ated with the validation of developed materials. 
The validation function includes assessing the valid- 
ity of each course elemerit, such ^a§: test items, 
lesson and module . instructioml riiaterials. The 
validity a^sessrrierit iricludes determiriirig the time 
required for accbnnpUishirig a learriirig objective iri 
relatiori to its cbritributibri to acquisitibri of pverall 
trairiirig bbjectives. This provides a deterriiiriatibri 
of course efficiericy^ iri addition tb learriirig effect-^ 
iveriess. 



Trairiirig Prbgrarini 
Validati bri 



Trairiirig program validaiion consists bf assessirig the 
exterit tb which the cbUrse as a whole meets the 
prescribed trairiirig bbjectives, iri terms bf actual job 
perfbrriiarice. this furictidn also serves to deter- 
riiirie the validity bf different courses or course 
cbrnpbrierits iri a seqUerice of cbUrses.^ A 
of this function in^yoives the determjnation of the 
cost effectiveness of the training system as a whole. 



Fuhrtior) 



Description 



RosbUrce Allocatiph 
Student Scheduling 
Instructor Scheduling 
Equipment Scheduling 
Equlprneht Inventory 



Instructional System Security 



Historical Record Keeping 



Resource allocation refers to those activities neces- 
sary for the scheduling of students, instructors, and 
equipment and supplies into the school to maintain 
an effective: balance of resources and optimal stu- 
dent flow through the course. A sub-function in- 
volves maintaining an inventory of equipment and 
supplies and handling the procurement process 
associated with assuring adequate available supplies. 

System security is the function that maintains the 
integrity of the Insr^ 'ictior xi system and consists of 
those activities the,i insure .laintaining the physical 
premises and the - ecrecy of test question-; and 
student records. For the basic school this fu'^ction 
may only involve tr - locking of an instructor's desk, 
while computer-based systems require passwords 
and computer sct:Urity instructions to keep tests and 
records and programs Trom being corhpromised. 

'Historical record keeping consists of thoseactivities 
necessary to maintain the desired records for evalu- 
ative purposes. This functibh is distinct frorh the 
particular evaluation activity itself. 
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